gh-128127: add feature for ast.NodeVisitor generic_visit to return list of all return values of calls to visit_

[ahmedhus22]

This implementation tries to ensure that previous behaviour of visit is not disturbed. For visit's new behaviour to work, derived classes need to call supers constructor, otherwise it behaves same as before.

• Issue: Allow ast.NodeVisitor's generic_visit to return list of all returned values of calls to visit_ #128127

[cpython-cla-bot]

All commit authors signed the Contributor License Agreement.

[bedevere-app]

Most changes to Python require a NEWS entry. Add one using the blurb_it web app or the blurb command-line tool.

If this change has little impact on Python users, wait for a maintainer to apply the skip news label instead.

[bedevere-app]

Most changes to Python require a NEWS entry. Add one using the blurb_it web app or the blurb command-line tool.

If this change has little impact on Python users, wait for a maintainer to apply the skip news label instead.

[picnixz]

This will blow-up memory footprint a lot, especially for static analysis tool so I'm not very sure it should be done. A visitor pattern is essentially used for traversing nodes, dynamically dispatching to the correct method efficiently, but it's not meant to remember what has been visited by default. If such behaviour is desired, a dedicated visitor class must be implemented instead.

[JelleZijlstra]

There's also some usability issues. For example, currently it's OK to reuse the same visitor instance for multiple concurrent or consecutive visits, but with this change, those runs will accumulate into the same list.

I'm not sure that collecting the return values of all visit_* calls in a single list is a very useful thing to do in the first place, since you don't preserve the structure of the AST.

[ahmedhus22]

This will blow-up memory footprint a lot, especially for static analysis tool so I'm not very sure it should be done. A visitor pattern is essentially used for traversing nodes, dynamically dispatching to the correct method efficiently, but it's not meant to remember what has been visited by default. If such behaviour is desired, a dedicated visitor class must be implemented instead.

I also thought about maybe this takes too much memory, and wasn't completely sure about this. Do you think I should make a new class for this feature or make tracking nodes optional(assuming this feature is still desirable)?

[ahmedhus22]

There's also some usability issues. For example, currently it's OK to reuse the same visitor instance for multiple concurrent or consecutive visits, but with this change, those runs will accumulate into the same list.

I'm not sure that collecting the return values of all visit_* calls in a single list is a very useful thing to do in the first place, since you don't preserve the structure of the AST.

• Oh, I guess the your first point means it should be optional (if still desirable)
• As for the usability, it was at least useful for me for getting some source code data about certain specific parts. So, it might be useful for gathering data about code.

[picnixz]

. Do you think I should make a new class for this feature or make tracking nodes optional(assuming this feature is still desirable)?

I don't think we'll accept this in the standard library, however you can create a third-party library and publish it on pypi.