Step 1: Setup GitHub REST API Requests 🤙

[nelsonic]

In order to get data from GitHub we need to setup the API.

According to the docs: https://docs.github.com/en/rest/overview/resources-in-the-rest-api 👀
the rate limiting for authenticated requests is 5,000/hour. 5️⃣ 🦘
That should be more than "enough" for what I have in mind. 📈
But if we need more we can easily add more API users. 💭

We've made HTTP Requests to the GitHub API before in previous projects:

• https://github.com/dwyl/elixir-auth-github
• https://github.com/dwyl/github-backup and
• https://github.com/nelsonic/issues

So I think I can piece this together from previous code. 🤞

Todo

• Setup GitHub API access and make a test request
• Use our existing knowledge/tools to fetch a list of @dwyl Org Members from the GitHub API

[nelsonic]

Decided to use https://github.com/edgurgel/tentacat for the GitHub API requests as it appears to be maintained. 👍

[nelsonic]

Used tentacat for basic requests and they worked ... 🎉
but then I re-read the GitHub REST API Docs:
https://docs.github.com/en/rest/orgs/members#list-organization-members

The GET /orgs/{org}/members request defaults to returning 30 members ...

@dwyl currently has 525 members: https://github.com/orgs/dwyl/people

We definitely need to paginate the request:

if we're unable to paginate the results this approach is a non-starter. 🤷‍♂️

Tentacat.Organizations.Members.list/2 function signature has only client and organisation arguments:
https://github.com/edgurgel/tentacat/blob/dfe89f33350acdf1ddea4a6f4d4338932fca0875/lib/tentacat/organizations/members.ex#L64-L67

 @doc """ 
 List members of a `organization`. 

 The response will differ if the authenticated user is also owner of the 
 organization. 

 ## Example 

     Tentacat.Organizations.Members.list "github" 
     Tentacat.Organizations.Members.list client, "github" 

 More info at: http://developer.github.com/v3/orgs/members/#members-list 
 """ 
 @spec list(Client.t(), binary) :: Tentacat.response() 
 def list(client \\ %Client{}, organization) do 
   get("orgs/#{organization}/members", client) 
 end 

Opened issue to confirm my understanding: edgurgel/tentacat#210 🤞

[nelsonic]

Might be able to use Tentacat.get/3 generic HTTP function to manually get the endpoint.

https://github.com/edgurgel/tentacat/blob/dfe89f33350acdf1ddea4a6f4d4338932fca0875/lib/tentacat.ex#L55-L95

@doc """
Underlying utility retrieval function. The options passed affect both the
return value and, ultimately, the number of requests made to GitHub.

## Options

  * `:pagination` - Can be `:none`, `:manual`, `:stream`, or `:auto`. Defaults to :auto.

      - `:none` will only return the first page. You won't have access to the
        headers to manually paginate.

      - `:auto` will block until all the pages have been retrieved and
        concatenated together. Most of the time, this is what you want. For
        example, `Tentacat.Repositories.list_users("chrismccord")` and
        `Tentacat.Repositories.list_users("octocat")` have the same interface
        though one call will page many times and the other not at all.

      - `:stream` will return a `Stream`, prepopulated with the first page.

      - `:manual` will return a 3 element tuple of `{page_body,
        url_for_next_page, auth_credentials}`, which will allow you to control
        the paging yourself.
"""
@spec get(binary, Client.t()) :: response
@spec get(binary, Client.t(), keyword) :: response
@spec get(binary, Client.t(), keyword, keyword) ::
        response | Enumerable.t() | pagination_response
def get(path, client, params \\ [], options \\ []) do
  url =
    client
    |> url(path)
    |> add_params_to_url(params)


  case pagination(options) do
    nil -> request_stream(:get, url, client.auth)
    :none -> request_stream(:get, url, client.auth, "", :one_page)
    :auto -> request_stream(:get, url, client.auth)
    :stream -> request_stream(:get, url, client.auth, "", :stream)
    :manual -> request_with_pagination(:get, url, client.auth)
  end
end

Thanks to edgurgel/tentacat#190 (comment) (comment on super-stale PR) 👀

Going to take a look at this on the Weekend. 🤞
Need to park this for now as it's lower priority than building the App/MVP. 😞

[nelsonic]

Going to try and cURL my way into figuring out the pagination ...

curl \
  -H "Accept: application/vnd.github+json" \
  -H "Authorization: Bearer <YOUR-TOKEN>" \
  https://api.github.com/orgs/dwyl/members

Again, one for Saturday. ⏳

[asntc]

Response:

[
  {
    "login": "alanshaw",
    "id": 152863,
    "node_id": "MDQ6VXNlcjE1Mjg2Mw==",
    "avatar_url": "https://avatars.githubusercontent.com/u/152863?v=4",
    "gravatar_id": "",
    "url": "https://api.github.com/users/alanshaw",
    "html_url": "https://github.com/alanshaw",
    "followers_url": "https://api.github.com/users/alanshaw/followers",
    "following_url": "https://api.github.com/users/alanshaw/following{/other_user}",
    "gists_url": "https://api.github.com/users/alanshaw/gists{/gist_id}",
    "starred_url": "https://api.github.com/users/alanshaw/starred{/owner}{/repo}",
    "subscriptions_url": "https://api.github.com/users/alanshaw/subscriptions",
    "organizations_url": "https://api.github.com/users/alanshaw/orgs",
    "repos_url": "https://api.github.com/users/alanshaw/repos",
    "events_url": "https://api.github.com/users/alanshaw/events{/privacy}",
    "received_events_url": "https://api.github.com/users/alanshaw/received_events",
    "type": "User",
    "user_view_type": "public",
    "site_admin": false
  }
]

[nelsonic]

Writing some Elixir code for this with @d4vcsj now. 🧑‍💻

[nelsonic]

App.User.get_user_from_api("edgurgel") #=>

{
  blog: "http://gurgel.me",
  location: "New Zealand",
  email: nil,
  avatar_url: "https://avatars.githubusercontent.com/u/30873?v=4",
  events_url: "https://api.github.com/users/edgurgel/events{/privacy}",
  public_repos: 32,
  updated_at: "2024-12-08T08:00:45Z",
  starred_url: "https://api.github.com/users/edgurgel/starred{/owner}{/repo}",
  company: nil,
  hireable: nil,
  repos_url: "https://api.github.com/users/edgurgel/repos",
  node_id: "MDQ6VXNlcjMwODcz",
  bio: "INSUFFICIENT DATA FOR MEANINGFUL ANSWER",
  organizations_url: "https://api.github.com/users/edgurgel/orgs",
  twitter_username: nil,
  user_view_type: "public",
  html_url: "https://github.com/edgurgel",
  followers_url: "https://api.github.com/users/edgurgel/followers",
  following_url: "https://api.github.com/users/edgurgel/following{/other_user}",
  received_events_url: "https://api.github.com/users/edgurgel/received_events",
  public_gists: 13,
  followers: 345,
  gists_url: "https://api.github.com/users/edgurgel/gists{/gist_id}",
  following: 67,
  gravatar_id: "",
  site_admin: false,
  login: "edgurgel",
  url: "https://api.github.com/users/edgurgel",
  id: 30873,
  name: "Eduardo Gurgel",
  created_at: "2008-10-24T17:05:04Z",
  type: "User",
  subscriptions_url: "https://api.github.com/users/edgurgel/subscriptions"
}

[nelsonic]

Tentacat.Organizations.Members.list/2 doesn't appear to have Pagination ...

https://github.com/edgurgel/tentacat/blob/b6b4d9fdeb50747d81af1f0aca9cb705558e565a/lib/tentacat/organizations/members.ex#L18-L21

  @spec list(Client.t(), binary) :: Tentacat.response()
  def list(client \\ %Client{}, organization) do
    get("orgs/#{organization}/members", client)
  end

Issue opened offering to add it: edgurgel/tentacat#219

[nelsonic]

PR: edgurgel/tentacat#220

[nelsonic]

Don't have to wait for the PR to be merged. ⏳
Can insert the first 30 records ... 🧑‍💻

[nelsonic]

Need to handle the 404 cases ...

1) test insert_org_memebers/1 (App.UserTest)
     test/app/user_test.exs:39
     ** (MatchError) no match of right hand side value: 
{404, %{"documentation_url" => "https://docs.github.com/rest", "message" => "Not Found", "status" => "404"}, 
%HTTPoison.Response{status_code: 404, body: %{"documentation_url" => "https://docs.github.com/rest", 
"message" => "Not Found", "status" => "404"}, headers: [{"Date", "Sun, 15 Dec 2024 08:54:54 GMT"}, 
{"Content-Type", "application/json; charset=utf-8"}, {"Content-Length", "89"}, {"X-OAuth-Scopes", "repo, user"}, 
{"X-Accepted-OAuth-Scopes", ""}, {"github-authentication-token-expiration", "2025-03-10 10:53:25 UTC"}, 
{"X-GitHub-Media-Type", "github.v3; format=json"}, {"x-github-api-version-selected", "2022-11-28"}, 
{"X-RateLimit-Limit", "5000"}, {"X-RateLimit-Remaining", "4450"}, {"X-RateLimit-Reset", "1734254887"}, 
{"X-RateLimit-Used", "550"}, {"X-RateLimit-Resource", "core"}, {"Access-Control-Expose-Headers", 
"ETag, Link, Location, Retry-After, X-GitHub-OTP, X-RateLimit-Limit, X-RateLimit-Remaining, 
X-RateLimit-Used, X-RateLimit-Resource, X-RateLimit-Reset, X-OAuth-Scopes, X-Accepted-OAuth-Scopes, 
X-Poll-Interval, X-GitHub-Media-Type, X-GitHub-SSO, X-GitHub-Request-Id, Deprecation, Sunset"},
 {"Access-Control-Allow-Origin", "*"}, {"Strict-Transport-Security", "max-age=31536000; includeSubdomains; preload"}, 
{"X-Frame-Options", "deny"}, {"X-Content-Type-Options", "nosniff"}, {"X-XSS-Protection", "0"}, 
{"Referrer-Policy", "origin-when-cross-origin, strict-origin-when-cross-origin"}, 
{"Content-Security-Policy", "default-src 'none'"}, {"Vary", "Accept-Encoding, Accept, X-Requested-With"}, 
{"Server", "github.com"}, {"X-GitHub-Request-Id", "D702:1E1B6C:595191B:5B6DF4B:675E995E"}], 
request_url: "https://api.github.com/users/kittenking", request: %HTTPoison.Request{method: :get, url: "https://api.github.com/users/kittenking", 
headers: [{"User-agent", "tentacat"}, {"Authorization", "token ghp_redacted"}], 
body: "\"\\\"\\\"\"", params: %{}, options: []}}}
     code: App.User.insert_org_memebers("dwyl")
     stacktrace:
       (app 1.7.0) lib/app/user.ex:57: App.User.get_user_from_api/1
       (elixir 1.17.3) lib/enum.ex:987: Enum."-each/2-lists^foreach/1-0-"/2
       test/app/user_test.exs:40: (test)

https://github.com/kittenking has made their profile private ... 🤦‍♂️

https://github.com/orgs/dwyl/people?query=kittenking

dunno if this is a mistake on his part ... but since it might happen again, need to add a condition for it. ⏳

[nelsonic]

PR edgurgel/tentacat#220 merged. ✅
Had to pause this to focus on other work. ⏸️ ⏳
But will take my Laptop to Xmas and squeeze in a bit of coding time. 🧑‍💻 🎄