Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Ruby/performance queries #18304

Draft
wants to merge 4 commits into
base: main
Choose a base branch
from
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 1 addition & 2 deletions ruby/ql/lib/codeql/ruby/frameworks/ActiveRecord.qll
Original file line number Diff line number Diff line change
Expand Up @@ -352,8 +352,7 @@
}

// A call to `find`, `where`, etc. that may return active record model object(s)
private class ActiveRecordModelFinderCall extends ActiveRecordModelInstantiation, DataFlow::CallNode
{
class ActiveRecordModelFinderCall extends ActiveRecordModelInstantiation, DataFlow::CallNode {

Check warning on line 355 in ruby/ql/lib/codeql/ruby/frameworks/ActiveRecord.qll

View workflow job for this annotation

GitHub Actions / qldoc

Missing QLdoc for class ActiveRecord::ActiveRecordModelFinderCall
private ActiveRecordModelClass cls;

ActiveRecordModelFinderCall() {
Expand Down
24 changes: 24 additions & 0 deletions ruby/ql/src/queries/performance/CouldBeAsync.qhelp
Original file line number Diff line number Diff line change
@@ -0,0 +1,24 @@
<!DOCTYPE qhelp PUBLIC
"-//Semmle//qhelp//EN"
"qhelp.dtd">
<qhelp>

<overview>
<p>
When an AcriveRecord query is executed synchronously, the application is blocked until the query completes. This can lead to poor performance, especially when the query is slow or when many queries are executed in sequence. This query identifies ActiveRecord queries that could be executed asynchronously to improve performance.
Specifically, this query identifies ActiveRecord queries that are executed in a loop. If each query is independent of the others, the queries could be executed in parallel by using the built-in Ruby <code>load_async</code> method.
In those cases, where the query includes a <code>pluck</code> call, the query could be executed asynchronously by using the <code>async_pluck</code> method.
</p>
</overview>
<recommendation>
<p>If possible, split the loop into two. The first creates a map of promises resolving to the async query results. The second runs throuhg this map, waiting on each promise, and does whatever the original loop did with the result of the query.</p>
</recommendation>
<example>
<p>The following (suboptimal) example code executes a series of ActiveRecord queries in a loop. The queries are independent of each other, so they could be executed in parallel to improve performance.</p>
<sample src="examples/straight_loop.rb" />
<p>To be able to fetch the necessary information asynchronously, we first pull it out into its own (implicit) loop:
<sample src="examples/preload.rb" />
<p>We can now use the <code>async_pluck</code> method to execute the queries in parallel.</p>
<sample src="examples/async_pluck.rb" />
</example>
</qhelp>
93 changes: 93 additions & 0 deletions ruby/ql/src/queries/performance/CouldBeAsync.ql
Original file line number Diff line number Diff line change
@@ -0,0 +1,93 @@
/**
* @name Could be async
* @description Use `ActiveRecord::Relation#load_async` to load records asynchronously.
* @kind problem
* @problem.severity info
* @precision high
* @id rb/could-be-async
* @tags performance
*/

import ruby
private import codeql.ruby.AST
import codeql.ruby.Concepts
import codeql.ruby.frameworks.ActiveRecord
private import codeql.ruby.TaintTracking

string loopMethodName() {
result in [
"each", "reverse_each", "map", "map!", "foreach", "flat_map", "in_batches", "one?", "all?",
"collect", "collect!", "select", "select!", "reject", "reject!"
]
}

class LoopingCall extends DataFlow::CallNode {
DataFlow::CallableNode loopBlock;

LoopingCall() {
this.getMethodName() = loopMethodName() and loopBlock = this.getBlock().asCallable()
}

DataFlow::CallableNode getLoopBlock() { result = loopBlock }
}

predicate happensInLoop(LoopingCall loop, DataFlow::CallNode e) {
loop.getLoopBlock().asCallableAstNode() = e.asExpr().getScope()
}

predicate happensInOuterLoop(LoopingCall outerLoop, DataFlow::CallNode e) {
exists(LoopingCall innerLoop |
happensInLoop(outerLoop, innerLoop) and
happensInLoop(innerLoop, e)
)
}

predicate happensInInnermostLoop(LoopingCall loop, DataFlow::CallNode e) {
happensInLoop(loop, e) and
not happensInOuterLoop(loop, e)
}

private class PluckCall extends ActiveRecordInstanceMethodCall {
PluckCall() { this.getMethodName() in ["pluck"] }

Check warning

Code scanning / CodeQL

Singleton set literal Warning

Singleton set literal can be replaced by its member.

ActiveRecordInstance chaines() { result = getChain(this) }
}

private ActiveRecordInstance getChain(ActiveRecordInstanceMethodCall c) {
result = c.getInstance()
or
result = getChain(c.getInstance())
}

// The ActiveRecord instance is used to potentially control the loop
predicate usedInLoopControlGuard(ActiveRecordInstance ar, DataFlow::Node guard) {
TaintTracking::localTaint(ar, guard) and
guard = guardForLoopControl(_, _)
}

// A guard for controlling the loop
DataFlow::Node guardForLoopControl(ConditionalExpr cond, Stmt control) {
result.asExpr().getAstNode() = cond.getCondition().getAChild*() and
(
control.(MethodCall).getMethodName() = "raise"
or
control instanceof NextStmt
) and
control = cond.getBranch(_).getAChild()
}

from LoopingCall loop, DataFlow::CallNode call, string message
where
not call.getLocation().getFile().getAbsolutePath().matches("%test%") and
not call = any(PluckCall p).chaines() and
not usedInLoopControlGuard(call, _) and
happensInInnermostLoop(loop, call) and
(
call instanceof ActiveRecordModelFinderCall and
not call.getMethodName() in ["new", "create"] and
message = "could be chained with load_async"
or
call instanceof PluckCall and
message = "could be async_pluck"
)
select call, "This call happens inside $@, and " + message, loop, "this loop"
16 changes: 16 additions & 0 deletions ruby/ql/src/queries/performance/examples/async_pluck.rb
Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@
require 'async_pluck'

# Preload User data in parallel
user_data = User.where(login: repo_names_by_owner.keys).async_pluck(:login, :id, :type).to_h do |login, id, type|
[login, { id: id, type: type == "User" ? "USER" : "ORGANIZATION" }]
end

repo_names_by_owner.each do |owner_slug, repo_names|
owner_info = user_data[owner_slug]
owner_id = owner_info[:id]
owner_type = owner_info[:type]
rel_conditions = { owner_id: owner_id, name: repo_names }

nwo_rel = nwo_rel.or(RepositorySecurityCenterConfig.where(rel_conditions)) unless neg
nwo_rel = nwo_rel.and(RepositorySecurityCenterConfig.where.not(rel_conditions)) if neg
end
14 changes: 14 additions & 0 deletions ruby/ql/src/queries/performance/examples/preload.rb
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
# Preload User data
user_data = User.where(login: repo_names_by_owner.keys).pluck(:login, :id, :type).to_h do |login, id, type|
[login, { id: id, type: type == "User" ? "USER" : "ORGANIZATION" }]
end

repo_names_by_owner.each do |owner_slug, repo_names|
owner_info = user_data[owner_slug]
owner_id = owner_info[:id]
owner_type = owner_info[:type]
rel_conditions = { owner_id: owner_id, name: repo_names }

nwo_rel = nwo_rel.or(RepositorySecurityCenterConfig.where(rel_conditions)) unless neg
nwo_rel = nwo_rel.and(RepositorySecurityCenterConfig.where.not(rel_conditions)) if neg
end
8 changes: 8 additions & 0 deletions ruby/ql/src/queries/performance/examples/straight_loop.rb
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
repo_names_by_owner.map do |owner_slug, repo_names|
owner_id, owner_type = User.where(login: owner_slug).pluck(:id, :type).first
owner_type = owner_type == "User" ? "USER" : "ORGANIZATION"
rel_conditions = { owner_id: owner_id, name: repo_names }

nwo_rel = nwo_rel.or(RepositorySecurityCenterConfig.where(rel_conditions)) unless neg
nwo_rel = nwo_rel.and(RepositorySecurityCenterConfig.where.not(rel_conditions)) if neg
end
Loading