Skip to content

YouTube Tutorial

Armin Darvish edited this page Apr 15, 2024 · 6 revisions

Announcing consult-web

Link to Video: https://www.youtube.com/watch?v=7pDfyqBZwvo

Web Search in browsers

Let’s take a look at web search in various browsers

FireFox

allows using different searhc engines and defining shortcuts

Chrome

allows using different searhc engines and defining shortcuts

Arc

  • allows using different searhc engines and defining shortcuts
  • has chatgpt and gmail as built-in search engines
  • allows doing instant open (simialr to Google’s I’m feeling Lucky)

Summary

  • Force you to pick a default search engine (instead of showing results from multiple sources)
  • Not much control over backends (e.g. which GenAI to use)
  • Limited control over UI, information presentation, sorting, …
  • Accessing previous search is not straightforward (you have to rely on suggestions or open history yourself)
  • Search results are not neccessarily reproducibile and there is no easy way to save the results

Web Search in Emacs

Let’s take a look at current available packages in Emacs.

Browsers inside Emacs

EWW

a minimal browser, mainly useful for plain text, but does not really add much new functionality to searching.

xwidget-webkit

A webkit (a.k.a. Safari) built-in in emacs. Everything else is the same as a normal browser (perhaps with just less security)

Send Queries to Search Engines

webjump

Webjump - WikEmacs is a simple built-in command that sends a query to search engines.

(setq webjump-sites
    `(("oogle" .
       [simple-query "www.google.com" "www.google.com/search?q=" ""])
      ("DuckDuckGo" .
       [simple-query "duckduckgo.com" "duckduckgo.com/?q=" ""])
      (" Wikipedia" .
       [simple-query "wikipedia.org" "wikipedia.org/wiki/" ""])
      (" StackOverflow" .
       [simple-query "www.stackoverflow.com"
                     "www.stackoverflow.com/search?q="
                     ""])
      (" Emacs Wiki" .
       [simple-query "www.emacswiki.org" "www.emacswiki.org/cgi-bin/wiki/" ""])
      (" github" .
       [simple-query "www.github.com" "https://github.com/search?q=" ""])
      (" Homepage"
       [simple-query "www.armindarvish.com" "https://www.armindarvish.com/q=" ""])
      ))

engine-mode

engine-mode is similar to webjump, but with more bells and whisltles.

(use-package engine-mode
    :straight t
    :config
    (defengine duckduckgo
    "https://duckduckgo.com/?q=%s"
    :term-transformation-hook upcase)
    (engine-mode t))

google-this-search

emacs-google-this is fesigned for searching thing at point

(use-package google-this
  :straight t)

Minibuffer Completion System

helm-google

emacs-helm · GitHub provides Google autosuggestion inside emacs.

counsel-web (with ivy+counsel)

counsel-web provides Google Search Results directly inside Emacs minibuffer, but because it uses HTML parsing your IP may get flagged if you hit the server too frequently.

Web Search in Emacs with consult-web

Get the package

For details, refer to the documentation on the Github repo here: consult-web. Maker sure to adjust the API key values for your own keys.

(use-package consult-web
:straight (consult-web :type git :host github :repo "armindarvish/consult-web" :branch "develop" :files (:defaults "extras/*.el" "sources/*.el"))
:after consult request elfeed
:custom
(consult-web-show-preview t)
(consult-web-preview-key "C-o")
(consult-web-highlight-matches t)
(consult-web-dynamic-input-debounce 0.8)
(consult-web-dynamic-input-throttle 1.6)
(consult-web-dynamic-refresh-delay 0.8)
:config
(require 'consult-web-sources)
(require 'consult-web-embark)
;; set api keys
(setq consult-web-google-customsearch-key "YOUR-GOOGLE-API-KEY"
      consult-web-google-customsearch-cx "YOUR-GOOGLE-CX-NUMBER"
      consult-web-brave-api-key "YOUR-BRAVE-API-KEY"
      consult-web-brave-autosuggest-api-key "YOUR-BRAVE-AUTOSUGGEST-API-KEY"
      consult-web-openai-api-key "YOUR-OPENAI-API-KEY"
      consult-web-stackexchange-api-key "YOUR-STACKEXCHANGE-API-KEY"
      consult-web-pubmed-api-key "YOUR-PUBMED-API-KEY"
      consult-web-bing-search-api-key "YOUR-BING-API-KEY"
      consult-web-scopus-api-key "YOUR-SCOPUS-API-KEY"
      )
)

Static Search

Simple Google Search

We can do google search by calling consult-web-google.

add autosuggestion

(setq consult-web-default-autosuggest-command #'consult-web-dynamic-brave-autosuggest)

Change Counts

(setq consult-web-default-count 10)

Use Brave Search

now let’s use Brave Search API by calling consult-web-brave

Change Grouping

no grouping

(setq consult-web-group-by nil)

custom grouping based on the url type

(defun ad/consult-web-group-test (cand transform)
  (when-let* ((url (get-text-property 0 :url cand))
              (urlobj (if url (url-generic-parse-url url) nil))
              (domain (if (url-p urlobj) (url-domain urlobj)))
              (name (cond
                     ((string-match ".*\\.gov.*\\|.*\\.uk.*\\|.*\\.eu.*" domain nil nil)
                      "Government")
                     ((string-match ".*\\.edu.*" domain nil nil)
                      "University")
                     ((member domain '("github.com"
                                       "gitlab.com"
                                       "bitbucket.org"
                                       "sourceforge.net"
                                       "sourcehut.org"
                                       "codeberg.org"
                                       ))
                      "Code")
                     ((member domain '("wikipedia.org"
                                       "dictionary.com"
                                       "merriam-webster.com"
                                       "britannica.com"))
                             "Encyclopedia")
                     ((member domain '("reddit.com"
                                       "stackexchange.com"
                                       "stackoverflow.com"
                                       "quora.com"))
                             "Community")
                     ((member domain '("orgmode.org"
                                       "gnu.org"
                                       "fsf.org"
                                       "emacswiki.org"
                                       "masteringemacs.org"))
                             "FOSS")
                     ((member domain '("spotify.com"
                                      "pandora.com"))
                             "Music")
                     ((member domain '("youtube.com"
                                      "vimeo.com"))
                             "Videos")
                     ((member domain '("nytimes.com"
                                      "washingtonpost.com"
                                      "cnn.com"
                                      "bbc.com"
                                      "economist.com"
                                      "apnews.com"
                                      "theguardian.com"))
                             "News")
                     (t
                      "Others"))))
  (if transform (substring cand) name)))

(setq consult-web-group-by #'ad/consult-web-group-test)

grouping by domain

(setq consult-web-group-by :domain)

Change Default Action

(setq consult-web-default-browse-function #'eww-browse-url)
(setq consult-web-default-browse-function #'org-web-tools-read-url-as-org)

use a custom-defined function

(setq ad/browsers
      (cl-remove nil `(,(if (functionp 'browse-url-arc) (cons "Arc" #'browse-url-arc))
                       ,(if (functionp 'browse-url-firefox) (cons "Firefox" #'browse-url-firefox))
                       ,(if (functionp 'browse-url-chrome) (cons "Chrome" #'browse-url-chrome))
                       ,(if (functionp 'eww-browse-url) (cons "EWW" #'eww-browse-url))
                       ,(if (functionp 'org-web-tools-read-url-as-org) (cons "Org-Web-Tools" #'org-web-tools-read-url-as-org))
                       ,(if (functionp 'browse-url-nyxt) (cons "Nyxt" #'browse-url-nyxt)))))

(defun ad/browse-url (&rest args)
  "Select the prefered browser from a menu before opening the URL."
  (interactive)
  (let ((browser (consult--read my:browsers
                                :prompt "Choose Browser: "
                                :require-match t))
        (args (or args (browse-url-interactive-arg "URL: "))))
    (apply (cdr (assoc browser my:browsers)) args)))

(setq consult-web-default-browse-function #'ad/browse-url)

What about AI?

simple chatgpt

we can use consult-web-chatgpt

gpel

We cna use the package gptel for a full-feature AI chat. Install gptel following the official documentation and then use consult-web-gptel.

Multi-Source Search

Let’s set the sources we want ot use:

(setq consult-web-default-count 5)
(setq consult-web-multi-sources '("gptel"
                                "Brave"
                                "Wikipedia"
                                "StackOverflow"
                                "YouTube"))

Now you can run the interactive command consult-web-multi

Dynamic Search

Dynamic Google Search

Let’s try consult-web-dynamic-google

Dynamic gptel and passing backends, models

Let’s see how passing arguments is useful for gptel consult-web-dynamic-gptel

Try entering the following in the minibuffer search (if you have acces to gpt-4!)

how to do web search in emacs -- --model gpt-4

Multi source and Dynamic

Let’ set the sources for multi-source dynamic search:

(setq consult-web-dynamic-sources '("gptel" "Brave" "Wikipedia" "StackOverflow" "YouTube"))

Now try the interactive command consult-web-dynamic.

Local Sources and Omni

consult-line-multi

Use consult-web-dynamic-consult-line-multi to see the consult-web version of consult-line-multi.

consult-notes

You need to install and setup consult-notes. Then try using consult-web-zettel-roam-nodes or the equivalent in your settings (the name of the function might be different depending on your consult-notes config).

elfeed

If you use elfeed, then try consult-web-dynamic-elfeed, otherwise first install and set up elfeed and then try the command.

consult-web-omni

Now let’s set sources for a dynamic multi-source “Omni” search (meaning both local and web sources)

(setq consult-web-dynamic-omni-sources (list "Known Project" "File" "Bookmark" "Buffer" "Reference Roam Nodes" "Zettel Roam Nodes" "Line Multi" "elfeed" "Brave" "Wikipedia" "gptel" "Youtube"))

Then try the interactive command consult-web-dynamic-omni!

Some More Sources

Searching Academic Literature

PubMed

Set up your pubmed account and API key. Then try the interactive command consult-web-dynamic-pubmed

consult-web-scholar

Set sources for multi-source academic literature search:

(setq consult-web-scholar-sources '("PubMed" "Scopus"))

Then try consult-web-scholar.

chnage the preview function to EWW (GNU Emacs Manual) ( or org-web-tools if available) and try reading an article

(setq consult-web-default-preview-function #'eww-browse-url)

Embark Actions

make custom actions

notes for academic literature

Here is an example for taking notes on academic literature. You can further connect this to org-capture templates as well.

(defun consult-web-embark-scholar-insert-note (cand)
  "insert note snippet for article"
  (let* ((url (and (stringp cand) (get-text-property 0 :url cand)))
         (url (and (stringp url) (string-trim url)))
         (doi (and (stringp cand) (get-text-property 0 :doi cand)))
         (doi (if (and doi (stringp doi)) (concat "https://doi.org/" doi)))
         (source (and (stringp cand) (get-text-property 0 :source cand)))
         (url (if (and (equal source "Scopus") doi)
                doi
                url))
         (title (and (stringp cand) (get-text-property 0 :title cand)))
         (authors (and (stringp cand) (get-text-property 0 :authors cand)))
         (authors (cond
                  ((and (listp authors) (= (length authors) 1))
                   (car authors))
                  ((listp authors)
                   (mapconcat #'identity authors ", "))
                  (t authors)))

         (journal  (and (stringp cand) (get-text-property 0 :journal cand)))
        (date (and (stringp cand) (get-text-property 0 :date cand))))

    (cond
      ((derived-mode-p 'org-mode)
       (insert (concat
                "\n"
                (cond
                 ((and url title) (format "** [[%s][%s]]\n" url title))
                 (url (format "** [[%s]]\n" url))
                 (title (format "** %s\n" title)))
                (if authors (format "\n%s" authors))
                (if journal (format "\nin =%s= " journal))
                (if date (format "published on [%s]\n" date) "\n")
                "\n*** Notes\n"
                )))
      ((derived-mode-p 'markdown-mode)
       (insert (concat
                "\n"
                (cond
                 ((and url title) (format "## [%s](%s)\n" url title))
                 (url (format "## <%s>\n" url))
                 (title (format "## %s\n" title)))
                (if authors (format "\n%s" authors))
                (if journal (format "\nin **%s** " journal))
                (if date (format "published on %s\n" date) "\n")
                "\n### Notes\n"
                )))
      (t
       (insert (concat
                "\n"
                (cond
                 ((and url title) (format "** %s (%s)\n" title  url))
                 (url (format "** %s\n" url))
                 (title (format "** %s\n" title)))
                (if authors (format "\n%s" authors))
                (if journal (format "\nin %s " journal))
                (if date (format "published on %s\n" date) "\n")
                "\n*** Notes\n"
                )))

      )))

(keymap-set consult-web-embark-scholar-actions-map "i n" #'consult-web-embark-scholar-insert-note)

open youtube in mpv

Here is an example for opening youtube videos with mpv.io.

(defun my:youtube-mpv (url &rest args)
(interactive (let* ((string (consult-web-dynamic-youtube nil t))
                    (url (and (stringp string) (get-text-property 0 :url string))))
               (list url)))
(if url
  (start-process "consult-web-mpv" nil "mpv"
                 url)))

(defun consult-web-embark-play-with-mpv (cand)
  (let* ((url (and (stringp cand) (get-text-property 0 :url cand))))
     (my:youtube-mpv url)))

;;;; if you have emacs package for mpv already installed
;; (defun consult-web-embark-play-with-mpv (cand)
;;    (let* ((url (and (stringp cand) (get-text-property 0 :url cand))))
;;      (mpv-play-url url)))

(keymap-set consult-web-embark-general-actions-map "o m" #'consult-web-embark-play-with-mpv)

Built-in actions per category

By default schoalr sources have a different category, but you can redefine the source and introduce new categories with their own embark actions,

Summarize Important Features

Minmial

Without docstrings and whitespaces the code is less than 1000 lines and it only depends on consult and buil-in url-retrieve.

Although adding other packages (embark, vertico, emacs-request, …) may help the overall workflow, they are not neccessary.

Modular

You can only load the parts you need. For example if all you need is an autosuggestion utility, then you can do:

(require 'consult-web-brave-autosuggest)

This adds an extra 100-200 lines of code per source.

Here is a table of all the currently implemented sources.

Source Category
chatGPT Simple AI prompts
Bing Search Engine
Brave Search Engine
Brave AutoSuggest AutoSuggest
consult-line-multi Local Text in Buffers
consult-notes Local Notes
consult-buffer Local Buffers
DuckDuckGo (Limited API) Search Suggestions
Elfeed Feeds (RSS, videos,…)
Google Search Engine
Google Autosuggest AutoSuggest
gptel AI Assistant
Doi.org Academic Reference
PubMed Academic Reference
Scopus Academic Reference
StackOverflow Community Forum
Wikipedia Encyclopedia
YouTube Videos

Customizable and Extendable

Lots of customization options. New sources can be added as you wish with different format, different actions,…

Power User capabilities

Dynamic collection allows for complex workflows on the fly. Change query parameters on the fly by passing arguments. Select a random set of results ad-hoc using embark and run embark actions on them. This allows batch processing as well. For example add a long list of sources for later review.

(defun consult-web-dwim (&optional input &rest args)
  (interactive)
  (let ((input (or input
                    (and consult-web-default-autosuggest-command (funcall-interactively consult-web-default-autosuggest-command))
                    (consult-web--read-search-string))))
(ignore-errors (minibuffer-with-setup-hook
    (lambda () (vertico-first) (embark-dwim))
  (consult-web-multi input args)))))