Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: github-org preset #267

Merged
merged 2 commits into from
Feb 11, 2022
Merged

feat: github-org preset #267

merged 2 commits into from
Feb 11, 2022

Conversation

breadadams
Copy link
Member

Related to #246

@vercel
Copy link

vercel bot commented Feb 1, 2022

This pull request is being automatically deployed with Vercel (learn more).
To see the status of your deployment, click below or on the icon next to each commit.

🔍 Inspect: https://vercel.com/microlink/cards/2Lv8gGiSgC2V8i8fPVtoHvGaz46Z
✅ Preview: https://cards-git-feat-github-org-preset-microlink.vercel.app

@breadadams
Copy link
Member Author

breadadams commented Feb 1, 2022

Hey @Kikobeats, this is the MQL query I mentioned I was having trouble with. Below you can see the elements I'm trying to scrape:

Problem solved with `prerender` option Screenshot 2022-02-01 at 22 37 42

When the page first loads, this data isn't there, and it's populated by JS:

image

I've tried with waitForTimeout and waitForSelector, but neither seem to work. This is a snippet to reproduce:

const github = () =>
  mql(`https://github.com/microlinkhq`, {
    // force: true,
    // waitForTimeout: 5000,
    // waitForSelector: 'a.topic-tag',
    data: {
      email: {
        selector: "a[itemprop=email]",
      },
      languages: {
        selectorAll: "a[href*='/repositories?language=']",
        attr: {
          color: {
            selector: ".repo-language-color",
            attr: "style",
          },
          language: {
            selector: "span[itemprop=programmingLanguage]",
          },
        },
      },
      description: {
        selector: "h1 + div > div",
      },
      tags: {
        selectorAll: ".topic-tag",
      },
      repositories: {
        selector: '.js-profile-tab-count-container a[href$="/repositories"] .Counter'
      },
      projects: {
        selector: '.js-profile-tab-count-container a[href*="/projects"] .Counter'
      },
      people: {
        selector: '.js-profile-tab-count-container a[href$="/people"] .Counter'
      },
    }
  })

This query yields the following result:

{
  title: 'microlink.io',
  description: 'Browser as API',
  lang: 'en',
  author: null,
  publisher: 'GitHub',
  image: {
    url: 'https://avatars.githubusercontent.com/u/29799436?s=280&v=4',
    type: 'png',
    size: 4118,
    height: 280,
    width: 280,
    size_pretty: '4.12 kB'
  },
  date: '2022-02-01T21:42:58.000Z',
  url: 'https://github.com/microlinkhq',
  logo: {
    url: 'https://github.com/fluidicon.png',
    type: 'png',
    size: 33270,
    height: 512,
    width: 512,
    size_pretty: '33.3 kB'
  },
  email: '[email protected]',
  languages: null,
  people: null,
  projects: null,
  repositories: null,
  tags: null
}

Any suggestions? Thanks

@Kikobeats
Copy link
Member

Thanks for the prerender explanation. I think at this time, since the data you are obtained there is very specific, prerender should be explicitly enabled. Happy to know after that it work fine 🙂

The preset looks terrific, such a great work.

@Kikobeats Kikobeats merged commit 9f53fbc into master Feb 11, 2022
@Kikobeats Kikobeats deleted the feat-github-org-preset branch February 11, 2022 09:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants