You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
If my assumption is correct, there are other simple workarounds besides
exporting the functions. Instead of putting
pkg:::foo(args)
into your script, put
pkg::callInternal("foo", args)
where pkg::callInternal is an exported function that can look up
unexported functions in the namespace.
Full message:
Message: 4
Date: Sun, 13 Sep 2020 16:19:46 -0400
From: Duncan Murdoch [...]
To: David Kepplinger [...], R Package Devel
[...]
Subject: Re: [R-pkg-devel] Use of `:::` in a package for code run in
a parallel cluster
Message-ID: <[email protected]>
Content-Type: text/plain; charset="utf-8"; Format="flowed"
On 13/09/2020 3:51 p.m., David Kepplinger wrote:
Dear list members,
I submitted an update for my package and got automatically rejected by the
incoming checks (as expected from my own checks) for using `:::` calls to
access the package's namespace.
"There are ::: calls to the package's namespace in its code. A package
*almost* never needs to use ::: for its own objects:…" (emphasis mine)
This was a conscious decision on my part as the package runs code on a
user-supplied parallel cluster and I consider cluster-exporting the
required functions a no-go as it would potentially overwrite objects in the
clusters R sessions. The package code does not own the cluster and hence
the R sessions. Therefore overwriting objects could potentially lead to
unintended behaviour which is opaque to the user and difficult to debug.
Another solution to circumvent the R CMD check note is to export the
functions to the public namespace but mark them as internal. This was also
suggested in another thread on this mailing list (c.f. "Etiquette for
package submissions that do not automatically pass checks?"). I do not
agree with this work-around as the methods are indeed internal and should
never be used by users. Exporting truly internal functions for the sake of
satisfying R CMD check is a bad argument, in particular if there is a
clean, well-documented, solution by using `:::`
Who is calling this function: package code or user code? I assume it's
a bit of a mix: your package writes a script that calls the function
when it runs in user space. (It would help if you gave an explicit
example of when you need to use this technique.)
If my assumption is correct, there are other simple workarounds besides
exporting the functions. Instead of putting
pkg:::foo(args)
into your script, put
pkg::callInternal("foo", args)
where pkg::callInternal is an exported function that can look up
unexported functions in the namespace.
You may argue that you prefer pkg:::foo for some reason: to which I'd
respond that you are being rude to the CRAN volunteers. I've offered
two options (one in the previous thread, a different one here), and
there was a third one in that thread offered by Ivan Krylov. Surely one
of these is good enough for your needs, and you shouldn't force CRAN to
handle you specially.
Duncan
The text was updated successfully, but these errors were encountered:
Send R-package-devel mailing list submissions to
[email protected]
To subscribe or unsubscribe via the World Wide Web, visit
https://stat.ethz.ch/mailman/listinfo/r-package-devel
or, via email, send a message with subject or body 'help' to
[email protected]
You can reach the person managing the list at
[email protected]
When replying, please edit your Subject line so it is more specific
than "Re: Contents of R-package-devel digest..."
Today's Topics:
1. Re: Use of `:::` in a package for code run in a parallel
cluster (Georgi Boshnakov)
2. Re: Use of `:::` in a package for code run in a parallel
cluster (Duncan Murdoch)
3. Re: Use of `:::` in a package for code run in a parallel
cluster (Wang, Zhu)
4. Re: Use of `:::` in a package for code run in a parallel
cluster (Jeff Newmiller)
5. Re: Use of `:::` in a package for code run in a parallel
cluster (Duncan Murdoch)
6. Re: Use of `:::` in a package for code run in a parallel
cluster (Wang, Zhu)
7. Re: Use of `:::` in a package for code run in a parallel
cluster (Duncan Murdoch)
8. Re: Use of `:::` in a package for code run in a parallel
cluster (David Kepplinger)
9. Re: Use of `:::` in a package for code run in a parallel
cluster (Henrik Bengtsson)
10. Re: Use of `:::` in a package for code run in a parallel
cluster (Cesko Voeten)
----------------------------------------------------------------------
Message: 1
Date: Mon, 14 Sep 2020 10:19:36 +0000
From: Georgi Boshnakov <[email protected]>
To: David Kepplinger <[email protected]>, R Package Devel
<[email protected]>
Subject: Re: [R-pkg-devel] Use of `:::` in a package for code run in
a parallel cluster
Message-ID:
<AM0PR0102MB34431FD43538A9E1D4069528AE230@AM0PR0102MB3443.eurprd01.prod.exchangelabs.com>
Content-Type: text/plain; charset="utf-8"
You may have a case to argue to CRAN that you can get the "almost" exemption (can't say without details) but your views look overly rigid.
Exporting an object and marking it as internal is not a "work around", even less a "dirty trick".
Export makes the object available outside the package's namespace and makes it clear that this is intentional.
If you can't drop the 'package:::' prefix in your use case, this means that this is what you actually do (i.e. use those objects outside the namespace of the package). I would be grateful to CRAN for asking me to export and hence document this.
Georgi Boshnakov
PS Note that there is no such thing as "public namespace".
-----Original Message-----
From: R-package-devel <[email protected]> On Behalf Of David Kepplinger
Sent: 13 September 2020 20:52
To: R Package Devel <[email protected]>
Subject: [R-pkg-devel] Use of `:::` in a package for code run in a parallel cluster
Dear list members,
I submitted an update for my package and got automatically rejected by the incoming checks (as expected from my own checks) for using `:::` calls to access the package's namespace.
"There are ::: calls to the package's namespace in its code. A package
*almost* never needs to use ::: for its own objects:…" (emphasis mine)
This was a conscious decision on my part as the package runs code on a user-supplied parallel cluster and I consider cluster-exporting the required functions a no-go as it would potentially overwrite objects in the clusters R sessions. The package code does not own the cluster and hence the R sessions. Therefore overwriting objects could potentially lead to unintended behaviour which is opaque to the user and difficult to debug.
Another solution to circumvent the R CMD check note is to export the functions to the public namespace but mark them as internal. This was also suggested in another thread on this mailing list (c.f. "Etiquette for package submissions that do not automatically pass checks?"). I do not agree with this work-around as the methods are indeed internal and should never be used by users. Exporting truly internal functions for the sake of satisfying R CMD check is a bad argument, in particular if there is a clean, well-documented, solution by using `:::`.
I argue `:::` is the only clean solution to this problem and no dirty work-arounds are necessary. This is a prime example of where `:::` is actually useful and needed inside a package. If the R community disagrees, I think R CMD check should at least emit a WARNING instead of a NOTE and elaborate on the problem and accepted work-arounds in "Writing R extensions". Or keep emitting a NOTE but listing those nebulous reasons where `:::` would be tolerated inside a package. Having more transparent criteria for submitting to CRAN would be really helpful to the entire R community and probably also reduce the traffic on this mailing list.
Best,
David
[[alternative HTML version deleted]]
______________________________________________
[email protected] mailing list https://stat.ethz.ch/mailman/listinfo/r-package-devel
------------------------------
Message: 2
Date: Mon, 14 Sep 2020 08:16:36 -0400
From: Duncan Murdoch <[email protected]>
To: "Wang, Zhu" <[email protected]>, David Kepplinger
<[email protected]>, R Package Devel
<[email protected]>
Subject: Re: [R-pkg-devel] Use of `:::` in a package for code run in
a parallel cluster
Message-ID: <[email protected]>
Content-Type: text/plain; charset="utf-8"; Format="flowed"
On 13/09/2020 8:47 p.m., Wang, Zhu wrote:
Apologize if I hijack this thread, but the use of ::: is something I was puzzled.
I tried Duncan's solution in my R package mypkg, something like:
pkg::callInternal("foo", args)
R CMD check mypkg
* checking dependencies in R code ... WARNING
'::' or ':::' import not declared from: ‘pkg'
I probably missed something here.
You don't need either pkg:: or pkg::: if you are calling the function
from within the package. You may need one of those if the call is
coming from a user script.
If you use pkg:: from mypkg, you need to declare that mypkg Imports pkg.
(This is a line in its DESCRIPTION file.) I think that's what the
WARNING is telling you.
Duncan Murdoch
Thanks,
Zhu
-----Original Message-----
From: R-package-devel <[email protected]> On Behalf Of Duncan Murdoch
Sent: Sunday, September 13, 2020 3:20 PM
To: David Kepplinger <[email protected]>; R Package Devel <[email protected]>
Subject: Re: [R-pkg-devel] Use of `:::` in a package for code run in a parallel cluster
On 13/09/2020 3:51 p.m., David Kepplinger wrote:
Dear list members,
I submitted an update for my package and got automatically rejected by
the incoming checks (as expected from my own checks) for using `:::`
calls to access the package's namespace.
"There are ::: calls to the package's namespace in its code. A package
*almost* never needs to use ::: for its own objects:…" (emphasis mine)
This was a conscious decision on my part as the package runs code on a
user-supplied parallel cluster and I consider cluster-exporting the
required functions a no-go as it would potentially overwrite objects
in the clusters R sessions. The package code does not own the cluster
and hence the R sessions. Therefore overwriting objects could
potentially lead to unintended behaviour which is opaque to the user and difficult to debug.
Another solution to circumvent the R CMD check note is to export the
functions to the public namespace but mark them as internal. This was
also suggested in another thread on this mailing list (c.f. "Etiquette
for package submissions that do not automatically pass checks?"). I do
not agree with this work-around as the methods are indeed internal and
should never be used by users. Exporting truly internal functions for
the sake of satisfying R CMD check is a bad argument, in particular if
there is a clean, well-documented, solution by using `:::`
Who is calling this function: package code or user code? I assume it's a bit of a mix: your package writes a script that calls the function when it runs in user space. (It would help if you gave an explicit example of when you need to use this technique.)
If my assumption is correct, there are other simple workarounds besides exporting the functions. Instead of putting
pkg:::foo(args)
into your script, put
pkg::callInternal("foo", args)
where pkg::callInternal is an exported function that can look up unexported functions in the namespace.
You may argue that you prefer pkg:::foo for some reason: to which I'd respond that you are being rude to the CRAN volunteers. I've offered two options (one in the previous thread, a different one here), and there was a third one in that thread offered by Ivan Krylov. Surely one of these is good enough for your needs, and you shouldn't force CRAN to handle you specially.
Duncan
I argue `:::` is the only clean solution to this problem and no dirty
work-arounds are necessary. This is a prime example of where `:::` is
actually useful and needed inside a package. If the R community
disagrees, I think R CMD check should at least emit a WARNING instead
of a NOTE and elaborate on the problem and accepted work-arounds in
"Writing R extensions". Or keep emitting a NOTE but listing those
nebulous reasons where `:::` would be tolerated inside a package.
Having more transparent criteria for submitting to CRAN would be
really helpful to the entire R community and probably also reduce the traffic on this mailing list.
Best,
David
[[alternative HTML version deleted]]
______________________________________________
[email protected] mailing list
https://stat.ethz.ch/mailman/listinfo/r-package-devel
______________________________________________
[email protected] mailing list https://stat.ethz.ch/mailman/listinfo/r-package-devel
------------------------------
Message: 3
Date: Mon, 14 Sep 2020 14:30:12 +0000
From: "Wang, Zhu" <[email protected]>
To: Duncan Murdoch <[email protected]>, David Kepplinger
<[email protected]>, R Package Devel
<[email protected]>
Subject: Re: [R-pkg-devel] Use of `:::` in a package for code run in
a parallel cluster
Message-ID:
<SN6PR10MB2432F38550F0989D2800437C9F230@SN6PR10MB2432.namprd10.prod.outlook.com>
Content-Type: text/plain; charset="utf-8"
In mypkg, I want to call a function foo from pkg, and foo is not exported. I thought I should use pkg:: or pkg:::, but R CMD check provided a warning.
Thanks,
Zhu
You don't need either pkg:: or pkg::: if you are calling the function from within the package. You may need one of those if the call is coming from a user script.
-----Original Message-----
From: Duncan Murdoch <[email protected]>
Sent: Monday, September 14, 2020 7:17 AM
To: Wang, Zhu <[email protected]>; David Kepplinger <[email protected]>; R Package Devel <[email protected]>
Subject: Re: [R-pkg-devel] Use of `:::` in a package for code run in a parallel cluster
On 13/09/2020 8:47 p.m., Wang, Zhu wrote:
Apologize if I hijack this thread, but the use of ::: is something I was puzzled.
I tried Duncan's solution in my R package mypkg, something like:
pkg::callInternal("foo", args)
R CMD check mypkg
* checking dependencies in R code ... WARNING '::' or ':::' import not
declared from: ‘pkg'
I probably missed something here.
You don't need either pkg:: or pkg::: if you are calling the function from within the package. You may need one of those if the call is coming from a user script.
If you use pkg:: from mypkg, you need to declare that mypkg Imports pkg.
(This is a line in its DESCRIPTION file.) I think that's what the WARNING is telling you.
Duncan Murdoch
Thanks,
Zhu
-----Original Message-----
From: R-package-devel <[email protected]> On
Behalf Of Duncan Murdoch
Sent: Sunday, September 13, 2020 3:20 PM
To: David Kepplinger <[email protected]>; R Package Devel
<[email protected]>
Subject: Re: [R-pkg-devel] Use of `:::` in a package for code run in a
parallel cluster
On 13/09/2020 3:51 p.m., David Kepplinger wrote:
Dear list members,
I submitted an update for my package and got automatically rejected
by the incoming checks (as expected from my own checks) for using
`:::` calls to access the package's namespace.
"There are ::: calls to the package's namespace in its code. A
package
*almost* never needs to use ::: for its own objects:…" (emphasis
mine)
This was a conscious decision on my part as the package runs code on
a user-supplied parallel cluster and I consider cluster-exporting the
required functions a no-go as it would potentially overwrite objects
in the clusters R sessions. The package code does not own the cluster
and hence the R sessions. Therefore overwriting objects could
potentially lead to unintended behaviour which is opaque to the user and difficult to debug.
Another solution to circumvent the R CMD check note is to export the
functions to the public namespace but mark them as internal. This was
also suggested in another thread on this mailing list (c.f.
"Etiquette for package submissions that do not automatically pass
checks?"). I do not agree with this work-around as the methods are
indeed internal and should never be used by users. Exporting truly
internal functions for the sake of satisfying R CMD check is a bad
argument, in particular if there is a clean, well-documented,
solution by using `:::`
Who is calling this function: package code or user code? I assume
it's a bit of a mix: your package writes a script that calls the
function when it runs in user space. (It would help if you gave an
explicit example of when you need to use this technique.)
If my assumption is correct, there are other simple workarounds
besides exporting the functions. Instead of putting
pkg:::foo(args)
into your script, put
pkg::callInternal("foo", args)
where pkg::callInternal is an exported function that can look up unexported functions in the namespace.
You may argue that you prefer pkg:::foo for some reason: to which I'd respond that you are being rude to the CRAN volunteers. I've offered two options (one in the previous thread, a different one here), and there was a third one in that thread offered by Ivan Krylov. Surely one of these is good enough for your needs, and you shouldn't force CRAN to handle you specially.
Duncan
I argue `:::` is the only clean solution to this problem and no dirty
work-arounds are necessary. This is a prime example of where `:::` is
actually useful and needed inside a package. If the R community
disagrees, I think R CMD check should at least emit a WARNING instead
of a NOTE and elaborate on the problem and accepted work-arounds in
"Writing R extensions". Or keep emitting a NOTE but listing those
nebulous reasons where `:::` would be tolerated inside a package.
Having more transparent criteria for submitting to CRAN would be
really helpful to the entire R community and probably also reduce the traffic on this mailing list.
Best,
David
[[alternative HTML version deleted]]
______________________________________________
[email protected] mailing list
https://stat.ethz.ch/mailman/listinfo/r-package-devel
______________________________________________
[email protected] mailing list
https://stat.ethz.ch/mailman/listinfo/r-package-devel
------------------------------
Message: 4
Date: Mon, 14 Sep 2020 08:06:32 -0700
From: Jeff Newmiller <[email protected]>
To: [email protected], "Wang, Zhu" <[email protected]>,
Duncan Murdoch <[email protected]>, David Kepplinger
<[email protected]>, R Package Devel
<[email protected]>
Subject: Re: [R-pkg-devel] Use of `:::` in a package for code run in
a parallel cluster
Message-ID: <[email protected]>
Content-Type: text/plain; charset="utf-8"
As Duncan said, if you are calling FROM a function in your package, and the function you are CALLING is in the same package, then you do NOT need any colons at all, whether exported or not.
On September 14, 2020 7:30:12 AM PDT, "Wang, Zhu" <[email protected]> wrote:
In mypkg, I want to call a function foo from pkg, and foo is not
exported. I thought I should use pkg:: or pkg:::, but R CMD check
provided a warning.
Thanks,
Zhu
You don't need either pkg:: or pkg::: if you are calling the function
from within the package. You may need one of those if the call is
coming from a user script.
-----Original Message-----
From: Duncan Murdoch <[email protected]>
Sent: Monday, September 14, 2020 7:17 AM
To: Wang, Zhu <[email protected]>; David Kepplinger
<[email protected]>; R Package Devel
<[email protected]>
Subject: Re: [R-pkg-devel] Use of `:::` in a package for code run in a
parallel cluster
On 13/09/2020 8:47 p.m., Wang, Zhu wrote:
Apologize if I hijack this thread, but the use of ::: is something I
was puzzled.
I tried Duncan's solution in my R package mypkg, something like:
pkg::callInternal("foo", args)
R CMD check mypkg
* checking dependencies in R code ... WARNING '::' or ':::' import
not
declared from: ‘pkg'
I probably missed something here.
You don't need either pkg:: or pkg::: if you are calling the function
from within the package. You may need one of those if the call is
coming from a user script.
If you use pkg:: from mypkg, you need to declare that mypkg Imports
pkg.
(This is a line in its DESCRIPTION file.) I think that's what the
WARNING is telling you.
Duncan Murdoch
Thanks,
Zhu
-----Original Message-----
From: R-package-devel <[email protected]> On
Behalf Of Duncan Murdoch
Sent: Sunday, September 13, 2020 3:20 PM
To: David Kepplinger <[email protected]>; R Package Devel
<[email protected]>
Subject: Re: [R-pkg-devel] Use of `:::` in a package for code run in
a
parallel cluster
On 13/09/2020 3:51 p.m., David Kepplinger wrote:
Dear list members,
I submitted an update for my package and got automatically rejected
by the incoming checks (as expected from my own checks) for using
`:::` calls to access the package's namespace.
"There are ::: calls to the package's namespace in its code. A
package
*almost* never needs to use ::: for its own objects:…" (emphasis
mine)
This was a conscious decision on my part as the package runs code on
a user-supplied parallel cluster and I consider cluster-exporting
the
required functions a no-go as it would potentially overwrite objects
in the clusters R sessions. The package code does not own the
cluster
and hence the R sessions. Therefore overwriting objects could
potentially lead to unintended behaviour which is opaque to the user
and difficult to debug.
Another solution to circumvent the R CMD check note is to export the
functions to the public namespace but mark them as internal. This
was
also suggested in another thread on this mailing list (c.f.
"Etiquette for package submissions that do not automatically pass
checks?"). I do not agree with this work-around as the methods are
indeed internal and should never be used by users. Exporting truly
internal functions for the sake of satisfying R CMD check is a bad
argument, in particular if there is a clean, well-documented,
solution by using `:::`
Who is calling this function: package code or user code? I assume
it's a bit of a mix: your package writes a script that calls the
function when it runs in user space. (It would help if you gave an
explicit example of when you need to use this technique.)
If my assumption is correct, there are other simple workarounds
besides exporting the functions. Instead of putting
pkg:::foo(args)
into your script, put
pkg::callInternal("foo", args)
where pkg::callInternal is an exported function that can look up
unexported functions in the namespace.
You may argue that you prefer pkg:::foo for some reason: to which
I'd respond that you are being rude to the CRAN volunteers. I've
offered two options (one in the previous thread, a different one here),
and there was a third one in that thread offered by Ivan Krylov.
Surely one of these is good enough for your needs, and you shouldn't
force CRAN to handle you specially.
Duncan
I argue `:::` is the only clean solution to this problem and no
dirty
work-arounds are necessary. This is a prime example of where `:::`
is
actually useful and needed inside a package. If the R community
disagrees, I think R CMD check should at least emit a WARNING
instead
of a NOTE and elaborate on the problem and accepted work-arounds in
"Writing R extensions". Or keep emitting a NOTE but listing those
nebulous reasons where `:::` would be tolerated inside a package.
Having more transparent criteria for submitting to CRAN would be
really helpful to the entire R community and probably also reduce
the traffic on this mailing list.
Best,
David
[[alternative HTML version deleted]]
______________________________________________
[email protected] mailing list
https://stat.ethz.ch/mailman/listinfo/r-package-devel
______________________________________________
[email protected] mailing list
https://stat.ethz.ch/mailman/listinfo/r-package-devel
______________________________________________
[email protected] mailing list
https://stat.ethz.ch/mailman/listinfo/r-package-devel
--
Sent from my phone. Please excuse my brevity.
------------------------------
Message: 5
Date: Mon, 14 Sep 2020 11:49:06 -0400
From: Duncan Murdoch <[email protected]>
To: "Wang, Zhu" <[email protected]>, David Kepplinger
<[email protected]>, R Package Devel
<[email protected]>
Subject: Re: [R-pkg-devel] Use of `:::` in a package for code run in
a parallel cluster
Message-ID: <[email protected]>
Content-Type: text/plain; charset="utf-8"; Format="flowed"
On 14/09/2020 10:30 a.m., Wang, Zhu wrote:
In mypkg, I want to call a function foo from pkg, and foo is not exported. I thought I should use pkg:: or pkg:::, but R CMD check provided a warning.
I'm assuming that mypkg is not the same as pkg; Jeff Newmiller's answer
assumes the opposite.
In that case where they are different, there is only one circumstance
where you should be calling foo, and you'll have to do it using
foo:::pkg. That circumstance is that you are the maintainer of both
packages. You should explain this in your submission message, and ask
CRAN to ignore the warning if there is one.
The reason for this is the following. If someone else is maintaining
pkg, then they are free to change the behaviour of foo without any
consideration for you, because as an internal function, they have no
contract with you to maintain its behaviour. On the other hand, if you
maintain both packages, then you should be ready to modify mypkg as soon
as you modify pkg:::foo.
Duncan Murdoch
Thanks,
Zhu
You don't need either pkg:: or pkg::: if you are calling the function from within the package. You may need one of those if the call is coming from a user script.
-----Original Message-----
From: Duncan Murdoch <[email protected]>
Sent: Monday, September 14, 2020 7:17 AM
To: Wang, Zhu <[email protected]>; David Kepplinger <[email protected]>; R Package Devel <[email protected]>
Subject: Re: [R-pkg-devel] Use of `:::` in a package for code run in a parallel cluster
On 13/09/2020 8:47 p.m., Wang, Zhu wrote:
Apologize if I hijack this thread, but the use of ::: is something I was puzzled.
I tried Duncan's solution in my R package mypkg, something like:
pkg::callInternal("foo", args)
R CMD check mypkg
* checking dependencies in R code ... WARNING '::' or ':::' import not
declared from: ‘pkg'
I probably missed something here.
You don't need either pkg:: or pkg::: if you are calling the function from within the package. You may need one of those if the call is coming from a user script.
If you use pkg:: from mypkg, you need to declare that mypkg Imports pkg.
(This is a line in its DESCRIPTION file.) I think that's what the WARNING is telling you.
Duncan Murdoch
Thanks,
Zhu
-----Original Message-----
From: R-package-devel <[email protected]> On
Behalf Of Duncan Murdoch
Sent: Sunday, September 13, 2020 3:20 PM
To: David Kepplinger <[email protected]>; R Package Devel
<[email protected]>
Subject: Re: [R-pkg-devel] Use of `:::` in a package for code run in a
parallel cluster
On 13/09/2020 3:51 p.m., David Kepplinger wrote:
Dear list members,
I submitted an update for my package and got automatically rejected
by the incoming checks (as expected from my own checks) for using
`:::` calls to access the package's namespace.
"There are ::: calls to the package's namespace in its code. A
package
*almost* never needs to use ::: for its own objects:…" (emphasis
mine)
This was a conscious decision on my part as the package runs code on
a user-supplied parallel cluster and I consider cluster-exporting the
required functions a no-go as it would potentially overwrite objects
in the clusters R sessions. The package code does not own the cluster
and hence the R sessions. Therefore overwriting objects could
potentially lead to unintended behaviour which is opaque to the user and difficult to debug.
Another solution to circumvent the R CMD check note is to export the
functions to the public namespace but mark them as internal. This was
also suggested in another thread on this mailing list (c.f.
"Etiquette for package submissions that do not automatically pass
checks?"). I do not agree with this work-around as the methods are
indeed internal and should never be used by users. Exporting truly
internal functions for the sake of satisfying R CMD check is a bad
argument, in particular if there is a clean, well-documented,
solution by using `:::`
Who is calling this function: package code or user code? I assume
it's a bit of a mix: your package writes a script that calls the
function when it runs in user space. (It would help if you gave an
explicit example of when you need to use this technique.)
If my assumption is correct, there are other simple workarounds
besides exporting the functions. Instead of putting
pkg:::foo(args)
into your script, put
pkg::callInternal("foo", args)
where pkg::callInternal is an exported function that can look up unexported functions in the namespace.
You may argue that you prefer pkg:::foo for some reason: to which I'd respond that you are being rude to the CRAN volunteers. I've offered two options (one in the previous thread, a different one here), and there was a third one in that thread offered by Ivan Krylov. Surely one of these is good enough for your needs, and you shouldn't force CRAN to handle you specially.
Duncan
I argue `:::` is the only clean solution to this problem and no dirty
work-arounds are necessary. This is a prime example of where `:::` is
actually useful and needed inside a package. If the R community
disagrees, I think R CMD check should at least emit a WARNING instead
of a NOTE and elaborate on the problem and accepted work-arounds in
"Writing R extensions". Or keep emitting a NOTE but listing those
nebulous reasons where `:::` would be tolerated inside a package.
Having more transparent criteria for submitting to CRAN would be
really helpful to the entire R community and probably also reduce the traffic on this mailing list.
Best,
David
[[alternative HTML version deleted]]
______________________________________________
[email protected] mailing list
https://stat.ethz.ch/mailman/listinfo/r-package-devel
______________________________________________
[email protected] mailing list
https://stat.ethz.ch/mailman/listinfo/r-package-devel
------------------------------
Message: 6
Date: Mon, 14 Sep 2020 15:56:20 +0000
From: "Wang, Zhu" <[email protected]>
To: Duncan Murdoch <[email protected]>, David Kepplinger
<[email protected]>, R Package Devel
<[email protected]>
Subject: Re: [R-pkg-devel] Use of `:::` in a package for code run in
a parallel cluster
Message-ID:
<SN6PR10MB24327128710B7095383E0EEF9F230@SN6PR10MB2432.namprd10.prod.outlook.com>
Content-Type: text/plain; charset="utf-8"
Yes, mypkg is different from pkg, and I am the maintainer of mypkg, but not pkg. Otherwise, things can be easier. Sorry for not clear enough.
Thanks to Duncan for a practical solution.
Best,
Zhu
-----Original Message-----
From: Duncan Murdoch <[email protected]>
Sent: Monday, September 14, 2020 10:49 AM
To: Wang, Zhu <[email protected]>; David Kepplinger <[email protected]>; R Package Devel <[email protected]>
Subject: Re: [R-pkg-devel] Use of `:::` in a package for code run in a parallel cluster
On 14/09/2020 10:30 a.m., Wang, Zhu wrote:
In mypkg, I want to call a function foo from pkg, and foo is not exported. I thought I should use pkg:: or pkg:::, but R CMD check provided a warning.
I'm assuming that mypkg is not the same as pkg; Jeff Newmiller's answer assumes the opposite.
In that case where they are different, there is only one circumstance where you should be calling foo, and you'll have to do it using foo:::pkg. That circumstance is that you are the maintainer of both packages. You should explain this in your submission message, and ask CRAN to ignore the warning if there is one.
The reason for this is the following. If someone else is maintaining pkg, then they are free to change the behaviour of foo without any consideration for you, because as an internal function, they have no contract with you to maintain its behaviour. On the other hand, if you maintain both packages, then you should be ready to modify mypkg as soon as you modify pkg:::foo.
Duncan Murdoch
Thanks,
Zhu
You don't need either pkg:: or pkg::: if you are calling the function from within the package. You may need one of those if the call is coming from a user script.
-----Original Message-----
From: Duncan Murdoch <[email protected]>
Sent: Monday, September 14, 2020 7:17 AM
To: Wang, Zhu <[email protected]>; David Kepplinger
<[email protected]>; R Package Devel
<[email protected]>
Subject: Re: [R-pkg-devel] Use of `:::` in a package for code run in a
parallel cluster
On 13/09/2020 8:47 p.m., Wang, Zhu wrote:
Apologize if I hijack this thread, but the use of ::: is something I was puzzled.
I tried Duncan's solution in my R package mypkg, something like:
pkg::callInternal("foo", args)
R CMD check mypkg
* checking dependencies in R code ... WARNING '::' or ':::' import
not declared from: ‘pkg'
I probably missed something here.
You don't need either pkg:: or pkg::: if you are calling the function from within the package. You may need one of those if the call is coming from a user script.
If you use pkg:: from mypkg, you need to declare that mypkg Imports pkg.
(This is a line in its DESCRIPTION file.) I think that's what the WARNING is telling you.
Duncan Murdoch
Thanks,
Zhu
-----Original Message-----
From: R-package-devel <[email protected]> On
Behalf Of Duncan Murdoch
Sent: Sunday, September 13, 2020 3:20 PM
To: David Kepplinger <[email protected]>; R Package Devel
<[email protected]>
Subject: Re: [R-pkg-devel] Use of `:::` in a package for code run in
a parallel cluster
On 13/09/2020 3:51 p.m., David Kepplinger wrote:
Dear list members,
I submitted an update for my package and got automatically rejected
by the incoming checks (as expected from my own checks) for using
`:::` calls to access the package's namespace.
"There are ::: calls to the package's namespace in its code. A
package
*almost* never needs to use ::: for its own objects:…" (emphasis
mine)
This was a conscious decision on my part as the package runs code on
a user-supplied parallel cluster and I consider cluster-exporting
the required functions a no-go as it would potentially overwrite
objects in the clusters R sessions. The package code does not own
the cluster and hence the R sessions. Therefore overwriting objects
could potentially lead to unintended behaviour which is opaque to the user and difficult to debug.
Another solution to circumvent the R CMD check note is to export the
functions to the public namespace but mark them as internal. This
was also suggested in another thread on this mailing list (c.f.
"Etiquette for package submissions that do not automatically pass
checks?"). I do not agree with this work-around as the methods are
indeed internal and should never be used by users. Exporting truly
internal functions for the sake of satisfying R CMD check is a bad
argument, in particular if there is a clean, well-documented,
solution by using `:::`
Who is calling this function: package code or user code? I assume
it's a bit of a mix: your package writes a script that calls the
function when it runs in user space. (It would help if you gave an
explicit example of when you need to use this technique.)
If my assumption is correct, there are other simple workarounds
besides exporting the functions. Instead of putting
pkg:::foo(args)
into your script, put
pkg::callInternal("foo", args)
where pkg::callInternal is an exported function that can look up unexported functions in the namespace.
You may argue that you prefer pkg:::foo for some reason: to which I'd respond that you are being rude to the CRAN volunteers. I've offered two options (one in the previous thread, a different one here), and there was a third one in that thread offered by Ivan Krylov. Surely one of these is good enough for your needs, and you shouldn't force CRAN to handle you specially.
Duncan
I argue `:::` is the only clean solution to this problem and no
dirty work-arounds are necessary. This is a prime example of where
`:::` is actually useful and needed inside a package. If the R
community disagrees, I think R CMD check should at least emit a
WARNING instead of a NOTE and elaborate on the problem and accepted
work-arounds in "Writing R extensions". Or keep emitting a NOTE but
listing those nebulous reasons where `:::` would be tolerated inside a package.
Having more transparent criteria for submitting to CRAN would be
really helpful to the entire R community and probably also reduce the traffic on this mailing list.
Best,
David
[[alternative HTML version deleted]]
______________________________________________
[email protected] mailing list
https://stat.ethz.ch/mailman/listinfo/r-package-devel
______________________________________________
[email protected] mailing list
https://stat.ethz.ch/mailman/listinfo/r-package-devel
------------------------------
Message: 7
Date: Mon, 14 Sep 2020 12:12:07 -0400
From: Duncan Murdoch <[email protected]>
To: "Wang, Zhu" <[email protected]>, David Kepplinger
<[email protected]>, R Package Devel
<[email protected]>
Subject: Re: [R-pkg-devel] Use of `:::` in a package for code run in
a parallel cluster
Message-ID: <[email protected]>
Content-Type: text/plain; charset="utf-8"; Format="flowed"
On 14/09/2020 11:56 a.m., Wang, Zhu wrote:
Yes, mypkg is different from pkg, and I am the maintainer of mypkg, but not pkg. Otherwise, things can be easier. Sorry for not clear enough.
Then you should not call foo, for the reasons I stated.
Alternatives are to contact the maintainer of pkg and explain why you
would like them to export foo, or (if the license permits it) just copy
the source of foo into your own package, giving appropriate credit to
the original author.
Duncan Murdoch
Thanks to Duncan for a practical solution.
Best,
Zhu
-----Original Message-----
From: Duncan Murdoch <[email protected]>
Sent: Monday, September 14, 2020 10:49 AM
To: Wang, Zhu <[email protected]>; David Kepplinger <[email protected]>; R Package Devel <[email protected]>
Subject: Re: [R-pkg-devel] Use of `:::` in a package for code run in a parallel cluster
On 14/09/2020 10:30 a.m., Wang, Zhu wrote:
In mypkg, I want to call a function foo from pkg, and foo is not exported. I thought I should use pkg:: or pkg:::, but R CMD check provided a warning.
I'm assuming that mypkg is not the same as pkg; Jeff Newmiller's answer assumes the opposite.
In that case where they are different, there is only one circumstance where you should be calling foo, and you'll have to do it using foo:::pkg. That circumstance is that you are the maintainer of both packages. You should explain this in your submission message, and ask CRAN to ignore the warning if there is one.
The reason for this is the following. If someone else is maintaining pkg, then they are free to change the behaviour of foo without any consideration for you, because as an internal function, they have no contract with you to maintain its behaviour. On the other hand, if you maintain both packages, then you should be ready to modify mypkg as soon as you modify pkg:::foo.
Duncan Murdoch
Thanks,
Zhu
You don't need either pkg:: or pkg::: if you are calling the function from within the package. You may need one of those if the call is coming from a user script.
-----Original Message-----
From: Duncan Murdoch <[email protected]>
Sent: Monday, September 14, 2020 7:17 AM
To: Wang, Zhu <[email protected]>; David Kepplinger
<[email protected]>; R Package Devel
<[email protected]>
Subject: Re: [R-pkg-devel] Use of `:::` in a package for code run in a
parallel cluster
On 13/09/2020 8:47 p.m., Wang, Zhu wrote:
Apologize if I hijack this thread, but the use of ::: is something I was puzzled.
I tried Duncan's solution in my R package mypkg, something like:
pkg::callInternal("foo", args)
R CMD check mypkg
* checking dependencies in R code ... WARNING '::' or ':::' import
not declared from: ‘pkg'
I probably missed something here.
You don't need either pkg:: or pkg::: if you are calling the function from within the package. You may need one of those if the call is coming from a user script.
If you use pkg:: from mypkg, you need to declare that mypkg Imports pkg.
(This is a line in its DESCRIPTION file.) I think that's what the WARNING is telling you.
Duncan Murdoch
Thanks,
Zhu
-----Original Message-----
From: R-package-devel <[email protected]> On
Behalf Of Duncan Murdoch
Sent: Sunday, September 13, 2020 3:20 PM
To: David Kepplinger <[email protected]>; R Package Devel
<[email protected]>
Subject: Re: [R-pkg-devel] Use of `:::` in a package for code run in
a parallel cluster
On 13/09/2020 3:51 p.m., David Kepplinger wrote:
Dear list members,
I submitted an update for my package and got automatically rejected
by the incoming checks (as expected from my own checks) for using
`:::` calls to access the package's namespace.
"There are ::: calls to the package's namespace in its code. A
package
*almost* never needs to use ::: for its own objects:…" (emphasis
mine)
This was a conscious decision on my part as the package runs code on
a user-supplied parallel cluster and I consider cluster-exporting
the required functions a no-go as it would potentially overwrite
objects in the clusters R sessions. The package code does not own
the cluster and hence the R sessions. Therefore overwriting objects
could potentially lead to unintended behaviour which is opaque to the user and difficult to debug.
Another solution to circumvent the R CMD check note is to export the
functions to the public namespace but mark them as internal. This
was also suggested in another thread on this mailing list (c.f.
"Etiquette for package submissions that do not automatically pass
checks?"). I do not agree with this work-around as the methods are
indeed internal and should never be used by users. Exporting truly
internal functions for the sake of satisfying R CMD check is a bad
argument, in particular if there is a clean, well-documented,
solution by using `:::`
Who is calling this function: package code or user code? I assume
it's a bit of a mix: your package writes a script that calls the
function when it runs in user space. (It would help if you gave an
explicit example of when you need to use this technique.)
If my assumption is correct, there are other simple workarounds
besides exporting the functions. Instead of putting
pkg:::foo(args)
into your script, put
pkg::callInternal("foo", args)
where pkg::callInternal is an exported function that can look up unexported functions in the namespace.
You may argue that you prefer pkg:::foo for some reason: to which I'd respond that you are being rude to the CRAN volunteers. I've offered two options (one in the previous thread, a different one here), and there was a third one in that thread offered by Ivan Krylov. Surely one of these is good enough for your needs, and you shouldn't force CRAN to handle you specially.
Duncan
I argue `:::` is the only clean solution to this problem and no
dirty work-arounds are necessary. This is a prime example of where
`:::` is actually useful and needed inside a package. If the R
community disagrees, I think R CMD check should at least emit a
WARNING instead of a NOTE and elaborate on the problem and accepted
work-arounds in "Writing R extensions". Or keep emitting a NOTE but
listing those nebulous reasons where `:::` would be tolerated inside a package.
Having more transparent criteria for submitting to CRAN would be
really helpful to the entire R community and probably also reduce the traffic on this mailing list.
Best,
David
[[alternative HTML version deleted]]
______________________________________________
[email protected] mailing list
https://stat.ethz.ch/mailman/listinfo/r-package-devel
______________________________________________
[email protected] mailing list
https://stat.ethz.ch/mailman/listinfo/r-package-devel
------------------------------
Message: 8
Date: Mon, 14 Sep 2020 12:06:08 -0700
From: David Kepplinger <[email protected]>
To: Georgi Boshnakov <[email protected]>
Cc: R Package Devel <[email protected]>
Subject: Re: [R-pkg-devel] Use of `:::` in a package for code run in
a parallel cluster
Message-ID:
<CAGmyFCGDJGAoVj86P-tE2oYAk28BqJczhYJi-qQk+mxQPcTrhQ@mail.gmail.com>
Content-Type: text/plain; charset="utf-8"
Yes, my view is certainly rigid and I agree that in the cases where the
function is actually used directly by the user, exporting it is the correct
step.
However, it seems some packages actually need to access internal functions
from an outside context, but the code that accesses the function is
logically contained completely inside the package. In these cases, package
maintainers seem to be looking for alternatives to `:::` for the sake of
avoiding the R CMD check note. I argue that the work arounds, however,
either (a) achieve the exact same result as `:::`, but in a less
transparent and likely more error prone way, or (b) unnecessarily making an
internal function available to the user.
I also agree with the CRAN team that package maintainers need to be made
aware of the issue when using `:::` inside their package as it is most
likely unnecessary. But the phrasing of the note ("almost never needs to
use :::") combined with a lack of transparent guidelines on when it is
acceptable leads to maintainers looking for alternatives mimicking the
behavior of `:::`. I haven't found any official instructions in Writing R
extensions or on the mailing list under what circumstances `:::` is deemed
to be acceptable by the CRAN team (I have to admit searching for `:::` in
the archives yields so many results I haven't looked at all of them). It's
probably impossible to conceive every possible use case for `:::`, but a
good start may be to have something in the documentation explicitly
mentioning commonly observed patterns where `:::` is not acceptable, and
the common exceptions to the rule (if there are any).
Maybe this issue is so miniscule and almost never comes up that it's not
worth mentioning in the documentation.
Best,
David
On Mon, Sep 14, 2020 at 3:19 AM Georgi Boshnakov <
[email protected]> wrote:
You may have a case to argue to CRAN that you can get the "almost"
exemption (can't say without details) but your views look overly rigid.
Exporting an object and marking it as internal is not a "work around",
even less a "dirty trick".
Export makes the object available outside the package's namespace and
makes it clear that this is intentional.
If you can't drop the 'package:::' prefix in your use case, this means
that this is what you actually do (i.e. use those objects outside the
namespace of the package). I would be grateful to CRAN for asking me to
export and hence document this.
Georgi Boshnakov
PS Note that there is no such thing as "public namespace".
-----Original Message-----
From: R-package-devel <[email protected]> On Behalf
Of David Kepplinger
Sent: 13 September 2020 20:52
To: R Package Devel <[email protected]>
Subject: [R-pkg-devel] Use of `:::` in a package for code run in a
parallel cluster
Dear list members,
I submitted an update for my package and got automatically rejected by the
incoming checks (as expected from my own checks) for using `:::` calls to
access the package's namespace.
"There are ::: calls to the package's namespace in its code. A package
*almost* never needs to use ::: for its own objects:…" (emphasis mine)
This was a conscious decision on my part as the package runs code on a
user-supplied parallel cluster and I consider cluster-exporting the
required functions a no-go as it would potentially overwrite objects in the
clusters R sessions. The package code does not own the cluster and hence
the R sessions. Therefore overwriting objects could potentially lead to
unintended behaviour which is opaque to the user and difficult to debug.
Another solution to circumvent the R CMD check note is to export the
functions to the public namespace but mark them as internal. This was also
suggested in another thread on this mailing list (c.f. "Etiquette for
package submissions that do not automatically pass checks?"). I do not
agree with this work-around as the methods are indeed internal and should
never be used by users. Exporting truly internal functions for the sake of
satisfying R CMD check is a bad argument, in particular if there is a
clean, well-documented, solution by using `:::`.
I argue `:::` is the only clean solution to this problem and no dirty
work-arounds are necessary. This is a prime example of where `:::` is
actually useful and needed inside a package. If the R community disagrees,
I think R CMD check should at least emit a WARNING instead of a NOTE and
elaborate on the problem and accepted work-arounds in "Writing R
extensions". Or keep emitting a NOTE but listing those nebulous reasons
where `:::` would be tolerated inside a package. Having more transparent
criteria for submitting to CRAN would be really helpful to the entire R
community and probably also reduce the traffic on this mailing list.
Best,
David
[[alternative HTML version deleted]]
______________________________________________
[email protected] mailing list
https://stat.ethz.ch/mailman/listinfo/r-package-devel
[[alternative HTML version deleted]]
------------------------------
Message: 9
Date: Mon, 14 Sep 2020 12:42:31 -0700
From: Henrik Bengtsson <[email protected]>
To: David Kepplinger <[email protected]>
Cc: Georgi Boshnakov <[email protected]>, R Package
Devel <[email protected]>
Subject: Re: [R-pkg-devel] Use of `:::` in a package for code run in
a parallel cluster
Message-ID:
<CAFDcVCQ+y6WX=3cWhR7zZF51fiZGUrLDG6qcj2E2T-HqMRmiBA@mail.gmail.com>
Content-Type: text/plain; charset="utf-8"
Without having read all of the comments already made here, but my
understanding why ::: is not allowed is because you are reaching into
the internal API that the package owner does not guarantee will exist
in the next release. If you rely on the internal code of another CRAN
package in your CRAN package, your CRAN package might break without
your control. This might release an avalanche of reverse package
dependencies failing on CRAN.
The only thing you can safely rely on is the API that is explicitly
*exported* by an R package. In order for the maintainer to break that
API for reverse dependent packages, they need to go through a process
of deprecating and defuncting what they want to break/remove - a
process that involves multiple releases and often reaching out to
package maintainers and asking them to update accordingly. CRAN runs
reverse package dependency checks making sure that a package does not
break its exported API. If it does, it will not roll out on CRAN.
So, in that sense CRAN helps uphold the contract of the exported APIs.
In contrast, a maintainer can do whatever they want whenever they want
with their internal code/API.
With more and more packages being infrastructure packages, I think
there is room for "protected" API, which is not exported to avoid
cluttering up the search path for end-users while it yet provides a
contract toward package developers relying on it. There are various
ways to emulate such protected APIs but we don't have a standard and
there's a risk that 'R CMD check' fails to detect when the contract is
broken (resulting in delayed run-time errors on the user end).
My $.02
Henrik
On Mon, Sep 14, 2020 at 12:06 PM David Kepplinger
<[email protected]> wrote:
Yes, my view is certainly rigid and I agree that in the cases where the
function is actually used directly by the user, exporting it is the correct
step.
However, it seems some packages actually need to access internal functions
from an outside context, but the code that accesses the function is
logically contained completely inside the package. In these cases, package
maintainers seem to be looking for alternatives to `:::` for the sake of
avoiding the R CMD check note. I argue that the work arounds, however,
either (a) achieve the exact same result as `:::`, but in a less
transparent and likely more error prone way, or (b) unnecessarily making an
internal function available to the user.
I also agree with the CRAN team that package maintainers need to be made
aware of the issue when using `:::` inside their package as it is most
likely unnecessary. But the phrasing of the note ("almost never needs to
use :::") combined with a lack of transparent guidelines on when it is
acceptable leads to maintainers looking for alternatives mimicking the
behavior of `:::`. I haven't found any official instructions in Writing R
extensions or on the mailing list under what circumstances `:::` is deemed
to be acceptable by the CRAN team (I have to admit searching for `:::` in
the archives yields so many results I haven't looked at all of them). It's
probably impossible to conceive every possible use case for `:::`, but a
good start may be to have something in the documentation explicitly
mentioning commonly observed patterns where `:::` is not acceptable, and
the common exceptions to the rule (if there are any).
Maybe this issue is so miniscule and almost never comes up that it's not
worth mentioning in the documentation.
Best,
David
On Mon, Sep 14, 2020 at 3:19 AM Georgi Boshnakov <
[email protected]> wrote:
You may have a case to argue to CRAN that you can get the "almost"
exemption (can't say without details) but your views look overly rigid.
Exporting an object and marking it as internal is not a "work around",
even less a "dirty trick".
Export makes the object available outside the package's namespace and
makes it clear that this is intentional.
If you can't drop the 'package:::' prefix in your use case, this means
that this is what you actually do (i.e. use those objects outside the
namespace of the package). I would be grateful to CRAN for asking me to
export and hence document this.
Georgi Boshnakov
PS Note that there is no such thing as "public namespace".
-----Original Message-----
From: R-package-devel <[email protected]> On Behalf
Of David Kepplinger
Sent: 13 September 2020 20:52
To: R Package Devel <[email protected]>
Subject: [R-pkg-devel] Use of `:::` in a package for code run in a
parallel cluster
Dear list members,
I submitted an update for my package and got automatically rejected by the
incoming checks (as expected from my own checks) for using `:::` calls to
access the package's namespace.
"There are ::: calls to the package's namespace in its code. A package
*almost* never needs to use ::: for its own objects:…" (emphasis mine)
This was a conscious decision on my part as the package runs code on a
user-supplied parallel cluster and I consider cluster-exporting the
required functions a no-go as it would potentially overwrite objects in the
clusters R sessions. The package code does not own the cluster and hence
the R sessions. Therefore overwriting objects could potentially lead to
unintended behaviour which is opaque to the user and difficult to debug.
Another solution to circumvent the R CMD check note is to export the
functions to the public namespace but mark them as internal. This was also
suggested in another thread on this mailing list (c.f. "Etiquette for
package submissions that do not automatically pass checks?"). I do not
agree with this work-around as the methods are indeed internal and should
never be used by users. Exporting truly internal functions for the sake of
satisfying R CMD check is a bad argument, in particular if there is a
clean, well-documented, solution by using `:::`.
I argue `:::` is the only clean solution to this problem and no dirty
work-arounds are necessary. This is a prime example of where `:::` is
actually useful and needed inside a package. If the R community disagrees,
I think R CMD check should at least emit a WARNING instead of a NOTE and
elaborate on the problem and accepted work-arounds in "Writing R
extensions". Or keep emitting a NOTE but listing those nebulous reasons
where `:::` would be tolerated inside a package. Having more transparent
criteria for submitting to CRAN would be really helpful to the entire R
community and probably also reduce the traffic on this mailing list.
Best,
David
[[alternative HTML version deleted]]
______________________________________________
[email protected] mailing list
https://stat.ethz.ch/mailman/listinfo/r-package-devel
[[alternative HTML version deleted]]
______________________________________________
[email protected] mailing list
https://stat.ethz.ch/mailman/listinfo/r-package-devel
------------------------------
Message: 10
Date: Tue, 15 Sep 2020 10:27:57 +0200
From: Cesko Voeten <[email protected]>
To: <[email protected]>
Subject: Re: [R-pkg-devel] Use of `:::` in a package for code run in
a parallel cluster
Message-ID: <[email protected]>
Content-Type: text/plain; charset="utf-8"; Format="flowed"
Henrik,
I completely agree with everything you wrote, but note that the issue at hand is using `:::' in *the same* package, for example when a package needs to access its own internal functions from an outside context, where running on a cluster node set up by the package is one thing I can think of. So there is no API contract to violate, except the one the package makes with itself. Given this, I'm inclined to agree with David: the language provides an obvious way to do this, why write a semantic kludge that is obviously intended only to circumvent the CRAN warning to achieve something that is necessary for the package? Of course, just my €.02 in a thought-provoking discussion!
Cesko
Op 14-09-2020 om 21:42 schreef Henrik Bengtsson:
Without having read all of the comments already made here, but my
understanding why ::: is not allowed is because you are reaching into
the internal API that the package owner does not guarantee will exist
in the next release. If you rely on the internal code of another CRAN
package in your CRAN package, your CRAN package might break without
your control. This might release an avalanche of reverse package
dependencies failing on CRAN.
The only thing you can safely rely on is the API that is explicitly
*exported* by an R package. In order for the maintainer to break that
API for reverse dependent packages, they need to go through a process
of deprecating and defuncting what they want to break/remove - a
process that involves multiple releases and often reaching out to
package maintainers and asking them to update accordingly. CRAN runs
reverse package dependency checks making sure that a package does not
break its exported API. If it does, it will not roll out on CRAN.
So, in that sense CRAN helps uphold the contract of the exported APIs.
In contrast, a maintainer can do whatever they want whenever they want
with their internal code/API.
With more and more packages being infrastructure packages, I think
there is room for "protected" API, which is not exported to avoid
cluttering up the search path for end-users while it yet provides a
contract toward package developers relying on it. There are various
ways to emulate such protected APIs but we don't have a standard and
there's a risk that 'R CMD check' fails to detect when the contract is
broken (resulting in delayed run-time errors on the user end).
My $.02
Henrik
On Mon, Sep 14, 2020 at 12:06 PM David Kepplinger
<[email protected]> wrote:
Yes, my view is certainly rigid and I agree that in the cases where the
function is actually used directly by the user, exporting it is the correct
step.
However, it seems some packages actually need to access internal functions
from an outside context, but the code that accesses the function is
logically contained completely inside the package. In these cases, package
maintainers seem to be looking for alternatives to `:::` for the sake of
avoiding the R CMD check note. I argue that the work arounds, however,
either (a) achieve the exact same result as `:::`, but in a less
transparent and likely more error prone way, or (b) unnecessarily making an
internal function available to the user.
I also agree with the CRAN team that package maintainers need to be made
aware of the issue when using `:::` inside their package as it is most
likely unnecessary. But the phrasing of the note ("almost never needs to
use :::") combined with a lack of transparent guidelines on when it is
acceptable leads to maintainers looking for alternatives mimicking the
behavior of `:::`. I haven't found any official instructions in Writing R
extensions or on the mailing list under what circumstances `:::` is deemed
to be acceptable by the CRAN team (I have to admit searching for `:::` in
the archives yields so many results I haven't looked at all of them). It's
probably impossible to conceive every possible use case for `:::`, but a
good start may be to have something in the documentation explicitly
mentioning commonly observed patterns where `:::` is not acceptable, and
the common exceptions to the rule (if there are any).
Maybe this issue is so miniscule and almost never comes up that it's not
worth mentioning in the documentation.
Best,
David
On Mon, Sep 14, 2020 at 3:19 AM Georgi Boshnakov <
[email protected]> wrote:
You may have a case to argue to CRAN that you can get the "almost"
exemption (can't say without details) but your views look overly rigid.
Exporting an object and marking it as internal is not a "work around",
even less a "dirty trick".
Export makes the object available outside the package's namespace and
makes it clear that this is intentional.
If you can't drop the 'package:::' prefix in your use case, this means
that this is what you actually do (i.e. use those objects outside the
namespace of the package). I would be grateful to CRAN for asking me to
export and hence document this.
Georgi Boshnakov
PS Note that there is no such thing as "public namespace".
-----Original Message-----
From: R-package-devel <[email protected]> On Behalf
Of David Kepplinger
Sent: 13 September 2020 20:52
To: R Package Devel <[email protected]>
Subject: [R-pkg-devel] Use of `:::` in a package for code run in a
parallel cluster
Dear list members,
I submitted an update for my package and got automatically rejected by the
incoming checks (as expected from my own checks) for using `:::` calls to
access the package's namespace.
"There are ::: calls to the package's namespace in its code. A package
*almost* never needs to use ::: for its own objects:…" (emphasis mine)
This was a conscious decision on my part as the package runs code on a
user-supplied parallel cluster and I consider cluster-exporting the
required functions a no-go as it would potentially overwrite objects in the
clusters R sessions. The package code does not own the cluster and hence
the R sessions. Therefore overwriting objects could potentially lead to
unintended behaviour which is opaque to the user and difficult to debug.
Another solution to circumvent the R CMD check note is to export the
functions to the public namespace but mark them as internal. This was also
suggested in another thread on this mailing list (c.f. "Etiquette for
package submissions that do not automatically pass checks?"). I do not
agree with this work-around as the methods are indeed internal and should
never be used by users. Exporting truly internal functions for the sake of
satisfying R CMD check is a bad argument, in particular if there is a
clean, well-documented, solution by using `:::`.
I argue `:::` is the only clean solution to this problem and no dirty
work-arounds are necessary. This is a prime example of where `:::` is
actually useful and needed inside a package. If the R community disagrees,
I think R CMD check should at least emit a WARNING instead of a NOTE and
elaborate on the problem and accepted work-arounds in "Writing R
extensions". Or keep emitting a NOTE but listing those nebulous reasons
where `:::` would be tolerated inside a package. Having more transparent
criteria for submitting to CRAN would be really helpful to the entire R
community and probably also reduce the traffic on this mailing list.
Best,
David
[[alternative HTML version deleted]]
______________________________________________
[email protected] mailing list
https://stat.ethz.ch/mailman/listinfo/r-package-devel
[[alternative HTML version deleted]]
______________________________________________
[email protected] mailing list
https://stat.ethz.ch/mailman/listinfo/r-package-devel
______________________________________________
[email protected] mailing list
https://stat.ethz.ch/mailman/listinfo/r-package-devel
------------------------------
Subject: Digest Footer
_______________________________________________
R-package-devel mailing list
[email protected]
https://stat.ethz.ch/mailman/listinfo/r-package-devel
------------------------------
End of R-package-devel Digest, Vol 65, Issue 14
***********************************************
Send R-package-devel mailing list submissions to
[email protected]
To subscribe or unsubscribe via the World Wide Web, visit
https://stat.ethz.ch/mailman/listinfo/r-package-devel
or, via email, send a message with subject or body 'help' to
[email protected]
You can reach the person managing the list at
[email protected]
When replying, please edit your Subject line so it is more specific
than "Re: Contents of R-package-devel digest..."
Today's Topics:
1. Re: Rcpp with clang++ -stdlib=libc++ ?
(=?UTF-8?Q?Dr=2E_Jens_Oehlschl=C3=A4gel?=)
2. CRAN incoming checks email (David Kepplinger)
3. Use of `:::` in a package for code run in a parallel cluster
(David Kepplinger)
4. Re: Use of `:::` in a package for code run in a parallel
cluster (Duncan Murdoch)
5. Re: Use of `:::` in a package for code run in a parallel
cluster (Martin Morgan)
6. Re: CRAN incoming checks email (Uwe Ligges)
7. Re: Use of `:::` in a package for code run in a parallel
cluster (Joshua Ulrich)
8. Re: Use of `:::` in a package for code run in a parallel
cluster (David Kepplinger)
9. Re: Use of `:::` in a package for code run in a parallel
cluster (Wang, Zhu)
----------------------------------------------------------------------
Message: 1
Date: Sun, 13 Sep 2020 16:04:56 +0200
From: =?UTF-8?Q?Dr=2E_Jens_Oehlschl=C3=A4gel?=
<[email protected]>
To: Kevin Ushey <[email protected]>, Dirk Eddelbuettel
<[email protected]>
Cc: R Package Development <[email protected]>
Subject: Re: [R-pkg-devel] Rcpp with clang++ -stdlib=libc++ ?
Message-ID:
<trinity-ced8b2f2-bbd9-4dd4-95c9-2d8f6c285f2a-1600005896629@3c-app-webde-bs09>
Content-Type: text/plain; charset="utf-8"
Thank you Dirk and Kevin,
that was very helpful and
sudo apt install libc++-dev libc++abi-dev
did the job!
Great, this was very important to me.
Thanks again
Jens
On 13.09.20 02:55, Kevin Ushey wrote:
My understanding is that many Linux OSes package the clang compiler, the
libc++ standard library, and the headers used by the libc++ standard
library separately. To install those headers, you likely need (e.g. on
Ubuntu):
sudo apt install libc++-dev libc++abi-dev
to be able to build and compile programs against libc++.
This also comes with the caveat that mixing programs built against
different standard library implementations is in general a bad idea, so
you may see issues if you mix libraries compiled with libstdc++ and
libc++ in the same R session. (This can come up with R packages that
link to other libraries installed on the system, which will typically be
built with and linked against the "default" system compiler + standard
library implementations.) I'm not sure if this will be an issue in
practice with what you're doing, but it's worth being aware of.
Best,
Kevin
On Sat, Sep 12, 2020 at 5:50 AM Dirk Eddelbuettel <[email protected]
<mailto:[email protected]>> wrote:
Hi Jens,
On 11 September 2020 at 21:00, Dr. Jens Oehlschlägel wrote:
| I can compile a package under clang++ with -stdlib=libstdc++, but
with -stdlib=libc++ I get
|
| "
| In file included from
/home/jo/R/x86_64-pc-linux-gnu-library/4.0/Rcpp/include/Rcpp/r/headers.h:67:
|
/home/jo/R/x86_64-pc-linux-gnu-library/4.0/Rcpp/include/Rcpp/platform/compiler.h:100:10:
fatal error: 'cmath' file not found
| #include <cmath>
| ^~~~~~~
| 1 error generated.
| "
|
| Is there any howto for using Rcpp with -stdlib=libc++ ?
That has zero to do with Rcpp. You are lacking a C++ library header
when
switching the C++ standard library along with clang. Nothing that
Rcpp ships,
or governs, or selects.
I am forgetting the fine details here (and someone may hopefully fill in
fuller details) but in short, "that is just the way it is". I think we
simply pivot back to the g++ standard C++ library even when using
clang++.
Cheers from Chicago, Dirk
| Greetings from Munich
|
| Jens Oehlschlägel
|
|
| P.S.
|
| Package Makevars
| CXX_STD = CXX17
| PKG_CXXFLAGS=-O3 -march=native -pthread
| PKG_LIBS=-latomic -pthread
|
| ~.R/Makevars
| CXX17 = clang++ -stdlib=libc++
| CXX17FLAGS = -fstack-protector-strong -D_FORTIFY_SOURCE=2 -g $(LTO)
| CXX17STD = -std=c++17
|
| > packageVersion("Rcpp")
| [1] ‘1.0.5’
|
| > version
| _
| platform x86_64-pc-linux-gnu
| arch x86_64
| os linux-gnu
| system x86_64, linux-gnu
| status
| major 4
| minor 0.2
| year 2020
| month 06
| day 22
| svn rev 78730
| language R
| version.string R version 4.0.2 (2020-06-22)
| nickname Taking Off Again
|
| ______________________________________________
| [email protected]
<mailto:[email protected]> mailing list
| https://stat.ethz.ch/mailman/listinfo/r-package-devel
--
https://dirk.eddelbuettel.com | @eddelbuettel | [email protected]
<mailto:[email protected]>
______________________________________________
[email protected] <mailto:[email protected]>
mailing list
https://stat.ethz.ch/mailman/listinfo/r-package-devel
------------------------------
Message: 2
Date: Sun, 13 Sep 2020 12:22:19 -0700
From: David Kepplinger <[email protected]>
To: R Package Devel <[email protected]>
Subject: [R-pkg-devel] CRAN incoming checks email
Message-ID:
<CAGmyFCGsE7zxs-kNt9MrkWoRONu9sL4f6=zGWK_f=-80WLUhxA@mail.gmail.com>
Content-Type: text/plain; charset="utf-8"
Dear List-Members:
The email from the automatic incoming checks says to "reply-all" in case
one suspects a false-positive, yet the reply-to header is set only to "
[email protected]". My email program (just as myself)
interprets this as "reply-all means replying only to
[email protected]". The wording in the email, on the other
hand, suggests I also should reply to Uwe Ligges. I find the current
disagreement of wording and email headers more than confusing.
Can someone clarify the correct protocol of replying to the email? I
wouldn't want to unnecessarily bother Uwe Ligges with even more emails.
Thanks,
David
[[alternative HTML version deleted]]
------------------------------
Message: 3
Date: Sun, 13 Sep 2020 12:51:41 -0700
From: David Kepplinger <[email protected]>
To: R Package Devel <[email protected]>
Subject: [R-pkg-devel] Use of `:::` in a package for code run in a
parallel cluster
Message-ID:
<CAGmyFCFVBXmyUR+YbOKB0Sc=KK67kYjQH18mewXDCTgGiNgQOg@mail.gmail.com>
Content-Type: text/plain; charset="utf-8"
Dear list members,
I submitted an update for my package and got automatically rejected by the
incoming checks (as expected from my own checks) for using `:::` calls to
access the package's namespace.
"There are ::: calls to the package's namespace in its code. A package
*almost* never needs to use ::: for its own objects:…" (emphasis mine)
This was a conscious decision on my part as the package runs code on a
user-supplied parallel cluster and I consider cluster-exporting the
required functions a no-go as it would potentially overwrite objects in the
clusters R sessions. The package code does not own the cluster and hence
the R sessions. Therefore overwriting objects could potentially lead to
unintended behaviour which is opaque to the user and difficult to debug.
Another solution to circumvent the R CMD check note is to export the
functions to the public namespace but mark them as internal. This was also
suggested in another thread on this mailing list (c.f. "Etiquette for
package submissions that do not automatically pass checks?"). I do not
agree with this work-around as the methods are indeed internal and should
never be used by users. Exporting truly internal functions for the sake of
satisfying R CMD check is a bad argument, in particular if there is a
clean, well-documented, solution by using `:::`.
I argue `:::` is the only clean solution to this problem and no dirty
work-arounds are necessary. This is a prime example of where `:::` is
actually useful and needed inside a package. If the R community disagrees,
I think R CMD check should at least emit a WARNING instead of a NOTE and
elaborate on the problem and accepted work-arounds in "Writing R
extensions". Or keep emitting a NOTE but listing those nebulous reasons
where `:::` would be tolerated inside a package. Having more transparent
criteria for submitting to CRAN would be really helpful to the entire R
community and probably also reduce the traffic on this mailing list.
Best,
David
[[alternative HTML version deleted]]
------------------------------
Message: 4
Date: Sun, 13 Sep 2020 16:19:46 -0400
From: Duncan Murdoch <[email protected]>
To: David Kepplinger <[email protected]>, R Package Devel
<[email protected]>
Subject: Re: [R-pkg-devel] Use of `:::` in a package for code run in
a parallel cluster
Message-ID: <[email protected]>
Content-Type: text/plain; charset="utf-8"; Format="flowed"
On 13/09/2020 3:51 p.m., David Kepplinger wrote:
Dear list members,
I submitted an update for my package and got automatically rejected by the
incoming checks (as expected from my own checks) for using `:::` calls to
access the package's namespace.
"There are ::: calls to the package's namespace in its code. A package
*almost* never needs to use ::: for its own objects:…" (emphasis mine)
This was a conscious decision on my part as the package runs code on a
user-supplied parallel cluster and I consider cluster-exporting the
required functions a no-go as it would potentially overwrite objects in the
clusters R sessions. The package code does not own the cluster and hence
the R sessions. Therefore overwriting objects could potentially lead to
unintended behaviour which is opaque to the user and difficult to debug.
Another solution to circumvent the R CMD check note is to export the
functions to the public namespace but mark them as internal. This was also
suggested in another thread on this mailing list (c.f. "Etiquette for
package submissions that do not automatically pass checks?"). I do not
agree with this work-around as the methods are indeed internal and should
never be used by users. Exporting truly internal functions for the sake of
satisfying R CMD check is a bad argument, in particular if there is a
clean, well-documented, solution by using `:::`
Who is calling this function: package code or user code? I assume it's
a bit of a mix: your package writes a script that calls the function
when it runs in user space. (It would help if you gave an explicit
example of when you need to use this technique.)
If my assumption is correct, there are other simple workarounds besides
exporting the functions. Instead of putting
pkg:::foo(args)
into your script, put
pkg::callInternal("foo", args)
where pkg::callInternal is an exported function that can look up
unexported functions in the namespace.
You may argue that you prefer pkg:::foo for some reason: to which I'd
respond that you are being rude to the CRAN volunteers. I've offered
two options (one in the previous thread, a different one here), and
there was a third one in that thread offered by Ivan Krylov. Surely one
of these is good enough for your needs, and you shouldn't force CRAN to
handle you specially.
Duncan
I argue `:::` is the only clean solution to this problem and no dirty
work-arounds are necessary. This is a prime example of where `:::` is
actually useful and needed inside a package. If the R community disagrees,
I think R CMD check should at least emit a WARNING instead of a NOTE and
elaborate on the problem and accepted work-arounds in "Writing R
extensions". Or keep emitting a NOTE but listing those nebulous reasons
where `:::` would be tolerated inside a package. Having more transparent
criteria for submitting to CRAN would be really helpful to the entire R
community and probably also reduce the traffic on this mailing list.
Best,
David
[[alternative HTML version deleted]]
______________________________________________
[email protected] mailing list
https://stat.ethz.ch/mailman/listinfo/r-package-devel
------------------------------
Message: 5
Date: Sun, 13 Sep 2020 22:04:54 +0000
From: Martin Morgan <[email protected]>
To: David Kepplinger <[email protected]>, R Package Devel
<[email protected]>
Subject: Re: [R-pkg-devel] Use of `:::` in a package for code run in
a parallel cluster
Message-ID:
<BL0PR04MB660968418CA6161F5D51234CF9220@BL0PR04MB6609.namprd04.prod.outlook.com>
Content-Type: text/plain; charset="utf-8"
At least in the 'parallel' package
library(parallel)
cl = makePSOCKcluster(2)
and because of the nature of the R language, the entire namespace is exported, analogous to
baz <- local({
foo <- function() 2
function(...) foo()
})
so making a package function baz available makes all functions in the package available -- a function in the package already has access to other functions in the namespace, whether those functions are exported or not, so there is no need to use :::.
parSapply(1:2, baz)
[1] 2 2
This is in contrast to what one might expect from exploring things on the command line, where foo is defined in the global environment and, by convention, the global environment is not serialized to the workers
foo <- function() 1
bar <- function(...) foo()
parLapply(cl, 1:2, bar)
Error in checkForRemoteErrors(val) :
2 nodes produced errors; first error: could not find function "foo"
Do you really need to use `:::`?
Martin Morgan
On 9/13/20, 3:52 PM, "R-package-devel on behalf of David Kepplinger" <[email protected] on behalf of [email protected]> wrote:
Dear list members,
I submitted an update for my package and got automatically rejected by the
incoming checks (as expected from my own checks) for using `:::` calls to
access the package's namespace.
"There are ::: calls to the package's namespace in its code. A package
*almost* never needs to use ::: for its own objects:…" (emphasis mine)
This was a conscious decision on my part as the package runs code on a
user-supplied parallel cluster and I consider cluster-exporting the
required functions a no-go as it would potentially overwrite objects in the
clusters R sessions. The package code does not own the cluster and hence
the R sessions. Therefore overwriting objects could potentially lead to
unintended behaviour which is opaque to the user and difficult to debug.
Another solution to circumvent the R CMD check note is to export the
functions to the public namespace but mark them as internal. This was also
suggested in another thread on this mailing list (c.f. "Etiquette for
package submissions that do not automatically pass checks?"). I do not
agree with this work-around as the methods are indeed internal and should
never be used by users. Exporting truly internal functions for the sake of
satisfying R CMD check is a bad argument, in particular if there is a
clean, well-documented, solution by using `:::`.
I argue `:::` is the only clean solution to this problem and no dirty
work-arounds are necessary. This is a prime example of where `:::` is
actually useful and needed inside a package. If the R community disagrees,
I think R CMD check should at least emit a WARNING instead of a NOTE and
elaborate on the problem and accepted work-arounds in "Writing R
extensions". Or keep emitting a NOTE but listing those nebulous reasons
where `:::` would be tolerated inside a package. Having more transparent
criteria for submitting to CRAN would be really helpful to the entire R
community and probably also reduce the traffic on this mailing list.
Best,
David
[[alternative HTML version deleted]]
______________________________________________
[email protected] mailing list
https://stat.ethz.ch/mailman/listinfo/r-package-devel
------------------------------
Message: 6
Date: Mon, 14 Sep 2020 00:05:16 +0200
From: Uwe Ligges <[email protected]>
To: David Kepplinger <[email protected]>, R Package Devel
<[email protected]>
Subject: Re: [R-pkg-devel] CRAN incoming checks email
Message-ID:
<[email protected]>
Content-Type: text/plain; charset="utf-8"; Format="flowed"
On 13.09.2020 21:22, David Kepplinger wrote:
Dear List-Members:
The email from the automatic incoming checks says to "reply-all" in case
one suspects a false-positive, yet the reply-to header is set only to "
[email protected]". My email program (just as myself)
interprets this as "reply-all means replying only to
[email protected]". The wording in the email, on the other
hand, suggests I also should reply to Uwe Ligges. I find the current
disagreement of wording and email headers more than confusing.
Can someone clarify the correct protocol of replying to the email? I
wouldn't want to unnecessarily bother Uwe Ligges with even more emails.
Thanks: Uwe has configured mails in a way he always receives one mail in
a CRAN-submissions folder for both cases.
Best,
Uwe
Thanks,
David
[[alternative HTML version deleted]]
______________________________________________
[email protected] mailing list
https://stat.ethz.ch/mailman/listinfo/r-package-devel
------------------------------
Message: 7
Date: Sun, 13 Sep 2020 17:32:13 -0500
From: Joshua Ulrich <[email protected]>
To: Duncan Murdoch <[email protected]>
Cc: David Kepplinger <[email protected]>, R Package Devel
<[email protected]>
Subject: Re: [R-pkg-devel] Use of `:::` in a package for code run in
a parallel cluster
Message-ID:
<CAPPM_gSC=7jUhdAtBie=po_pQBuRDFY0mrpifa+RRLPzHrp94A@mail.gmail.com>
Content-Type: text/plain; charset="utf-8"
On Sun, Sep 13, 2020 at 3:19 PM Duncan Murdoch <[email protected]> wrote:
On 13/09/2020 3:51 p.m., David Kepplinger wrote:
Dear list members,
I submitted an update for my package and got automatically rejected by the
incoming checks (as expected from my own checks) for using `:::` calls to
access the package's namespace.
"There are ::: calls to the package's namespace in its code. A package
*almost* never needs to use ::: for its own objects:…" (emphasis mine)
This was a conscious decision on my part as the package runs code on a
user-supplied parallel cluster and I consider cluster-exporting the
required functions a no-go as it would potentially overwrite objects in the
clusters R sessions. The package code does not own the cluster and hence
the R sessions. Therefore overwriting objects could potentially lead to
unintended behaviour which is opaque to the user and difficult to debug.
Another solution to circumvent the R CMD check note is to export the
functions to the public namespace but mark them as internal. This was also
suggested in another thread on this mailing list (c.f. "Etiquette for
package submissions that do not automatically pass checks?"). I do not
agree with this work-around as the methods are indeed internal and should
never be used by users. Exporting truly internal functions for the sake of
satisfying R CMD check is a bad argument, in particular if there is a
clean, well-documented, solution by using `:::`
Who is calling this function: package code or user code? I assume it's
a bit of a mix: your package writes a script that calls the function
when it runs in user space. (It would help if you gave an explicit
example of when you need to use this technique.)
If my assumption is correct, there are other simple workarounds besides
exporting the functions. Instead of putting
pkg:::foo(args)
into your script, put
pkg::callInternal("foo", args)
where pkg::callInternal is an exported function that can look up
unexported functions in the namespace.
Another possibility is what quantmod::newTA() does.
https://github.com/joshuaulrich/quantmod/blob/a8e9cb87825c0997a8468f5105db6c507b26ac5d/R/newTA.
It's a function that creates a user-facing function. The created
function needs to access unexported objects from the quantmod
namespace. newTA() accomplishes that by setting the environment of
the function it returns to the quantmod namespace.
https://github.com/joshuaulrich/quantmod/blob/a8e9cb87825c0997a8468f5105db6c507b26ac5d/R/newTA.R#L98
That gives the user's new function access to the unexported charting
objects it needs to work.
Hope that helps.
Best,
Josh
You may argue that you prefer pkg:::foo for some reason: to which I'd
respond that you are being rude to the CRAN volunteers. I've offered
two options (one in the previous thread, a different one here), and
there was a third one in that thread offered by Ivan Krylov. Surely one
of these is good enough for your needs, and you shouldn't force CRAN to
handle you specially.
Duncan
I argue `:::` is the only clean solution to this problem and no dirty
work-arounds are necessary. This is a prime example of where `:::` is
actually useful and needed inside a package. If the R community disagrees,
I think R CMD check should at least emit a WARNING instead of a NOTE and
elaborate on the problem and accepted work-arounds in "Writing R
extensions". Or keep emitting a NOTE but listing those nebulous reasons
where `:::` would be tolerated inside a package. Having more transparent
criteria for submitting to CRAN would be really helpful to the entire R
community and probably also reduce the traffic on this mailing list.
Best,
David
[[alternative HTML version deleted]]
______________________________________________
[email protected] mailing list
https://stat.ethz.ch/mailman/listinfo/r-package-devel
______________________________________________
[email protected] mailing list
https://stat.ethz.ch/mailman/listinfo/r-package-devel
--
Joshua Ulrich | about.me/joshuaulrich
FOSS Trading | www.fosstrading.com
------------------------------
Message: 8
Date: Sun, 13 Sep 2020 17:00:46 -0700
From: David Kepplinger <[email protected]>
To: R Package Devel <[email protected]>
Subject: Re: [R-pkg-devel] Use of `:::` in a package for code run in
a parallel cluster
Message-ID:
<CAGmyFCEUtz-VnaMCYRKk=01qWGT3pADgacQ-mmYKTmUmW_E6fw@mail.gmail.com>
Content-Type: text/plain; charset="utf-8"
Thank you all for the discussion and suggestions.
so making a package function baz available makes all functions in the
package available -- a function in the package already has access to other
functions in the namespace, whether those functions are exported or not, so
there is no need to use :::.
Thanks, Martin. I completely missed that the parallel package serializes
the entire environment of the function, including the package namespace and
so `:::` is indeed unnecessary in my use case. I probably experimented in
the global environment first and extrapolated the observed behaviour to the
package. Sorry for annoying everyone with this.
I also have another use of `:::` for which I am not sure if it's considered
disallowed use of `:::`, so I'm throwing it out there for feedback.
I have one internal function which checks a long list of common arguments
to several other functions, similar to
internal_check_args <- function (sd = 2, type = c("bootstrap",
"theoretical"), ...) {
# check arguments for valid ranges, etc.
return(list(sd = sd, type = match.arg(type))
}
And several functions which use the internal function for argument checking
and such, similar to
exported_foo <- function (x, sd = 2, type = c("bootstrap", "theoretical")) {
args_call <- match.call()
args_call[[1]] <- quote(mypackage:::internal_check_args)
args <- eval.parent(args_call)
}
exported_foo_cv <- function (x, cv_folds = 3, ...) {
args_call <- match.call(expand.dots = TRUE)
args_call[[1]] <- quote(mypackage:::internal_check_args)
args <- eval.parent(args_call)
}
This is modelled after what, e.g., `lm()` does with `model.frame()`, only
that `internal_check_args()` is not exported, hence I use `:::`. There are
other solutions for this type of use of `:::` (probably some considered
cleaner) but again without guidelines on when `:::` is acceptable it's
difficult for package maintainers to know when to use/not use it. From all
the discussions it seems that there is absolutely no acceptable use of
`:::` and work-arounds are always the better alternative.
In light of the other interesting points brought up by discussants, I also
want to honor their time and reply here.
You may argue that you prefer pkg:::foo for some reason: to which I'd
respond that you are being rude to the CRAN volunteers. I've offered
two options (one in the previous thread, a different one here), and
there was a third one in that thread offered by Ivan Krylov. Surely one
of these is good enough for your needs, and you shouldn't force CRAN to
handle you specially.
I am sorry it came across rude when I tried to solicit arguments for why
the use of `:::` is considered "bad practice", while work-arounds are
considered to be okay. I wouldn't force CRAN to handle my case specially; I
rather wanted to challenge the general "attitude" towards the use of `:::`.
I am sure there is a need to discourage the use of `:::` in packages as
CRAN volunteers probably have seen hundreds of cases where `:::` was abused
(such as mine, as Martin Morgan pointed out).
you can use
get("internal_function", asNamespace("mypackage"))(arg1, arg2)
In fact, if you look at the source code of `:::`, that's exactly how
it is implemented:
That would be a work-around I would have used if necessary. But my general
question remains: why should I reinvent the wheel when R already comes with
`:::`? The only advantage of all the work-arounds I've seen would be to
trick R CMD check that the code is okay, when in fact the same "bad
practice" is practiced.
Best,
David
On Sun, Sep 13, 2020 at 3:04 PM Martin Morgan <[email protected]>
wrote:
At least in the 'parallel' package
library(parallel)
cl = makePSOCKcluster(2)
and because of the nature of the R language, the entire namespace is
exported, analogous to
baz <- local({
foo <- function() 2
function(...) foo()
})
so making a package function baz available makes all functions in the
package available -- a function in the package already has access to other
functions in the namespace, whether those functions are exported or not, so
there is no need to use :::.
parSapply(1:2, baz)
[1] 2 2
This is in contrast to what one might expect from exploring things on the
command line, where foo is defined in the global environment and, by
convention, the global environment is not serialized to the workers
foo <- function() 1
bar <- function(...) foo()
parLapply(cl, 1:2, bar)
Error in checkForRemoteErrors(val) :
2 nodes produced errors; first error: could not find function "foo"
Do you really need to use `:::`?
Martin Morgan
On 9/13/20, 3:52 PM, "R-package-devel on behalf of David Kepplinger" <
[email protected] on behalf of
[email protected]> wrote:
Dear list members,
I submitted an update for my package and got automatically rejected by
the
incoming checks (as expected from my own checks) for using `:::` calls
to
access the package's namespace.
"There are ::: calls to the package's namespace in its code. A package
*almost* never needs to use ::: for its own objects:…" (emphasis mine)
This was a conscious decision on my part as the package runs code on a
user-supplied parallel cluster and I consider cluster-exporting the
required functions a no-go as it would potentially overwrite objects
in the
clusters R sessions. The package code does not own the cluster and
hence
the R sessions. Therefore overwriting objects could potentially lead to
unintended behaviour which is opaque to the user and difficult to
debug.
Another solution to circumvent the R CMD check note is to export the
functions to the public namespace but mark them as internal. This was
also
suggested in another thread on this mailing list (c.f. "Etiquette for
package submissions that do not automatically pass checks?"). I do not
agree with this work-around as the methods are indeed internal and
should
never be used by users. Exporting truly internal functions for the
sake of
satisfying R CMD check is a bad argument, in particular if there is a
clean, well-documented, solution by using `:::`.
I argue `:::` is the only clean solution to this problem and no dirty
work-arounds are necessary. This is a prime example of where `:::` is
actually useful and needed inside a package. If the R community
disagrees,
I think R CMD check should at least emit a WARNING instead of a NOTE
and
elaborate on the problem and accepted work-arounds in "Writing R
extensions". Or keep emitting a NOTE but listing those nebulous reasons
where `:::` would be tolerated inside a package. Having more
transparent
criteria for submitting to CRAN would be really helpful to the entire R
community and probably also reduce the traffic on this mailing list.
Best,
David
[[alternative HTML version deleted]]
______________________________________________
[email protected] mailing list
https://stat.ethz.ch/mailman/listinfo/r-package-devel
[[alternative HTML version deleted]]
------------------------------
Message: 9
Date: Mon, 14 Sep 2020 00:47:22 +0000
From: "Wang, Zhu" <[email protected]>
To: Duncan Murdoch <[email protected]>, David Kepplinger
<[email protected]>, R Package Devel
<[email protected]>
Subject: Re: [R-pkg-devel] Use of `:::` in a package for code run in
a parallel cluster
Message-ID:
<SN6PR10MB24320CB8AA24B1A1DA6BBA8A9F230@SN6PR10MB2432.namprd10.prod.outlook.com>
Content-Type: text/plain; charset="utf-8"
Apologize if I hijack this thread, but the use of ::: is something I was puzzled.
I tried Duncan's solution in my R package mypkg, something like:
pkg::callInternal("foo", args)
R CMD check mypkg
* checking dependencies in R code ... WARNING
'::' or ':::' import not declared from: ‘pkg'
I probably missed something here.
Thanks,
Zhu
-----Original Message-----
From: R-package-devel <[email protected]> On Behalf Of Duncan Murdoch
Sent: Sunday, September 13, 2020 3:20 PM
To: David Kepplinger <[email protected]>; R Package Devel <[email protected]>
Subject: Re: [R-pkg-devel] Use of `:::` in a package for code run in a parallel cluster
On 13/09/2020 3:51 p.m., David Kepplinger wrote:
Dear list members,
I submitted an update for my package and got automatically rejected by
the incoming checks (as expected from my own checks) for using `:::`
calls to access the package's namespace.
"There are ::: calls to the package's namespace in its code. A package
*almost* never needs to use ::: for its own objects:…" (emphasis mine)
This was a conscious decision on my part as the package runs code on a
user-supplied parallel cluster and I consider cluster-exporting the
required functions a no-go as it would potentially overwrite objects
in the clusters R sessions. The package code does not own the cluster
and hence the R sessions. Therefore overwriting objects could
potentially lead to unintended behaviour which is opaque to the user and difficult to debug.
Another solution to circumvent the R CMD check note is to export the
functions to the public namespace but mark them as internal. This was
also suggested in another thread on this mailing list (c.f. "Etiquette
for package submissions that do not automatically pass checks?"). I do
not agree with this work-around as the methods are indeed internal and
should never be used by users. Exporting truly internal functions for
the sake of satisfying R CMD check is a bad argument, in particular if
there is a clean, well-documented, solution by using `:::`
Who is calling this function: package code or user code? I assume it's a bit of a mix: your package writes a script that calls the function when it runs in user space. (It would help if you gave an explicit example of when you need to use this technique.)
If my assumption is correct, there are other simple workarounds besides exporting the functions. Instead of putting
pkg:::foo(args)
into your script, put
pkg::callInternal("foo", args)
where pkg::callInternal is an exported function that can look up unexported functions in the namespace.
You may argue that you prefer pkg:::foo for some reason: to which I'd respond that you are being rude to the CRAN volunteers. I've offered two options (one in the previous thread, a different one here), and there was a third one in that thread offered by Ivan Krylov. Surely one of these is good enough for your needs, and you shouldn't force CRAN to handle you specially.
Duncan
I argue `:::` is the only clean solution to this problem and no dirty
work-arounds are necessary. This is a prime example of where `:::` is
actually useful and needed inside a package. If the R community
disagrees, I think R CMD check should at least emit a WARNING instead
of a NOTE and elaborate on the problem and accepted work-arounds in
"Writing R extensions". Or keep emitting a NOTE but listing those
nebulous reasons where `:::` would be tolerated inside a package.
Having more transparent criteria for submitting to CRAN would be
really helpful to the entire R community and probably also reduce the traffic on this mailing list.
Best,
David
[[alternative HTML version deleted]]
______________________________________________
[email protected] mailing list
https://stat.ethz.ch/mailman/listinfo/r-package-devel
______________________________________________
[email protected] mailing list https://stat.ethz.ch/mailman/listinfo/r-package-devel
------------------------------
Subject: Digest Footer
_______________________________________________
R-package-devel mailing list
[email protected]
https://stat.ethz.ch/mailman/listinfo/r-package-devel
------------------------------
End of R-package-devel Digest, Vol 65, Issue 13
***********************************************
From Duncan Murdoch:
Full message:
The text was updated successfully, but these errors were encountered: