Online editing

[qaisjp]

A long term plan is to get editing right within the website, rather than just on GitHub.

gollum / gollum-lib seems like a really good base, but this will not be an easy feat.

[qaisjp]

So on July 28 I contacted Informatics Computing Services about getting authentication behind Informatics' Cosign service.

Today we've gotten the OK at first sight, and they are going to discuss this next week (person managing the Cosign service is off work this week). This is looking positive!

[qaisjp]

We have the Cosign service working in a quick telnet-esque test. Here's the game plan:

• move content to a simple folder structure not running on jekyll (this would be the "content" repo)

then, one of the following:

A) write a new website in Ruby (with the use of gollum) to edit the content repo
B) Keep with jekyll, but add client side authentication (+ write an API) with client side editing, and so on

[qaisjp]

This is going pretty well!

I've written a library to interact with the CoSign daemon. Anything living on Tardis can interact with cosign-webapi instead.

Proof of concept of CoSign working can be found here, on the map! (Log in on the map before visiting the next link..)

A Better Informatics "API", of sorts, can be found at provider.betterinformatics.com (https only). The code for bi-provider is open-source, of course!

[rphi]

That Cosign thing is very neat indeed....

I started typing things and realised I have no idea what I'm talking about RE editing, but I'd probably vote for having the site remain on Jekyll. Alternatively and an idea that you haven't already mentioned, is maybe even serve the site content as Markdown files direct from the repository and then use a client side library to parse the MD on the fly. Then for editing, just click to swap to the raw MD. How you'd get the changes back to Github is another question entirely and would probably need something awful.

[Visgean]

I think the way to go here is simply use commits signed by the cosign user.

[rphi]

Oh of course, because in Git there is no authentication on who you say you are in the commits is there? So we could give the back-end an API key which it could use for all users' edits?

Also just come across this that might come in handy: Github Contents API

[Visgean]

You dont need that, you can simply commit the changes and sign it with the cosign email.

…

On Tue, 7 Nov 2017, 20:30 Robert Phipps, ***@***.***> wrote: Oh of course, because in Git there is no authentication on who you say you are in the commits is there? So we could give the back-end an API key which it could use for all users' edits? Also just come across this that might come in handy: Github Contents API <https://developer.github.com/v3/repos/contents/> — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#23 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAL1KIwQv9XA2XYu10X0LFXMmPJJFI09ks5s0L31gaJpZM4OSMTx> .

[rphi]

But to push to the repository you need to authenticate?

[rphi]

(assuming this still needs you to be on the CompSoc Github team to push)

[qaisjp]

I'm hoping for us to integrate editing within the Better Informatics website.

It shouldn't be just 3-5 people contributing content, and people are much more likely to contribute if they can click a big edit button and make a change immediately, rather than having to create a GitHub accounts. Unfortunately a lot of students don't know how to use Git.

Right now my plan is:

• First write a small form that will link your DICE account to GitHub, automating the GitHub team process.
• Do the same thing for Google Drive, as the Team Drive is private unless you request access. (We have been receiving requests.) done!
• Do a /my page, showing your courses, as it should be fairly simple. (bi-provider is one half of that) Update: this has been done but we have a problem with LDAP access
• Then write an interface of edit file + commit, where editing in the browser will send the change to the server, and create a commit for you. (Which is sent back to GitHub.)
• Then move away from Jekyll, but keep the edit file + commit system with GitHub, as I'm fairly keen on leveraging Git for version control.

Moving away from Jekyll would be ideal as we don't have that many pages, and we have a lot of metadata (about courses) that could be pulled externally (such as what semester courses are in, what courses are available, course names, etc).

But even when moving away from Jekyll, all the real content would still be stored in markdown files, visible on GitHub.

[qaisjp]

We can use GitHub Apps.

The Git Database API gives you access to read and write raw Git objects to your Git database on GitHub and to list and update your references (branch heads and tags).

This basically allows you to reimplement a lot of Git functionality over our API - by creating raw objects directly into the database and updating branch references you could technically do just about anything that Git can do without having Git installed.

This makes this feature a whole lot more possible! It means we don't have to deal with races between the bi.tardis.ed.ac.uk and the GitHub copy of our repository.

The cool thing here is that we can create commits as users themselves (potentially from the client).

Because trees must be created before creating commits we could actually support editing multiple files in one go (although we probably technically don't want this as this would result in non-gitty commits, but that ship has probably sailed).

The interface could look like this (just a quick idea and a quick drawing):

After clicking "save" we can show them a diff and ask them to write a commit title/message.

We can ask them if they have a GitHub account or not and either:

• Using user-to-server requests: add them to the org, create a commit as them, using their default github email (potential to give them a list if they use multiple emails)
• Just create a commit as the app (as committer) using their SMS email (as author).

The advantages of user-to-server requests here aren't clear, as we can create a commit as them whether or not they like it. Adding them to the organisation would be nice, though.

[penalosa]

So a possible future direction for this is Netlify's NetlifyCMS. Their CMS itself is a fairly opinionated React app, but as part of that they've open sourced a proxy to the GitHub API that allows for client side commits to be made, which we could integrate into an editing thing inline on BI. I'll have a play around and see what the feasibility of this might be, but it could be a good way forward fo getting more people contributing.

[penalosa]

TLDR

Proof of concept is now up at bi.comp-soc.com

In more words

This has been a bit of a saga, and has entailed some fairly fundamental changes to a lot of betterinformatics, in a way that might not necessarily want to be taken forward — it's something to discuss.

In essence, what I've done is split out the engine of BI from the content. The content still lives in the betterinformatics repo, while the engine now lives in bi-engine. There were a couple of reasons for this; to ease integration of online editing by moving away from Jekyll as the engine, and to facilitate extension of BI into other subject areas. The engine is written in Svelte, and is essentially a straight clone of the Jekyll site (Svelte's templating syntax is quite close to Jekyll's). There are probably still some rough edges, but it's pretty much feature complete (with exactly the same styling).

On to the exciting stuff — online editing! The general flow is:

• Edit on the client in markdown (using CodeMirror). Editing the front matter is explicitly not supported, because that should be a fairly rare case, and should be done by a maintainer, ideally.
• Send the new file over to a Cloudflare Worker, with makes the necessary GitHub API calls (proxied through a git-gateway instance that will be replaced with a GitHub app when I have time) to edit that file and commit the changes
• GitHub Actions pulls in the latest engine version and builds the site to the gh-pages branch
• This was served by GitHub Pages, but given their pretty harsh caching, it's now on Netlify, which has much faster updates.

Any thoughts? This is a fairly bare description, but I can expand on any bits of it that are unclear. I'm of the mind that this direction would get more contributions, because it would be drastically easier to contribute, but I'm aware that it does entail a fairly sweeping change.

(CC @compsoc-edinburgh/sigweb)

[qaisjp]

In essence, what I've done is split out the engine of BI from the content. The content still lives in the betterinformatics repo, while the engine now lives in bi-engine.

👍 👍 👍 💯 💯

Love it.

This was served by GitHub Pages, but given their pretty harsh caching, it's now on Netlify, which has much faster updates.

cool! How does that work? Just it just serve from the gh-pages branch (from which repo?)

or does it do away with gh-pages altogether (i guess netlify handles the building too)?

Edit on the client in markdown (using CodeMirror).

BI has an open source license for https://ckeditor.com/ laying around if we want to add WYSIWYG support later.

WYSIWYG can be quite complicated to do nicely though (especially rendering TO markdown) so perhaps we should not look into this right now and exclude it from the 'minimum viable product'.

(i hate the MVP acronym.)

Just putting this out there for documentation sake!

Editing the front matter is explicitly not supported, because that should be a fairly rare case, and should be done by a maintainer, ideally.

Yeah. Most things, year: 2 tag included, can be sourced from the courses data file. Some things like the 'quick links' might be useful for others to edit in the webface, though.

Shouldn't be too hard to read the front matter (it's just yaml after all), merge in the user-provided 'quick links data', and write it back out to the front matter?

Though, as above, front matter editing does not seem necessary for the first iteration.

Send the new file over to a Cloudflare Worker, with makes the necessary GitHub API calls (proxied through a git-gateway instance that will be replaced with a GitHub app when I have time) to edit that file and commit the changes

Cool! Is this the frontend connecting to the backend here? https://github.com/compsoc-edinburgh/bi-engine/blob/master/save-section/index.js

Where's the sauce for this? (Or is this what you're planning to commit when you're free? no pressure)

necessary GitHub API calls

(I'm assuming source for the git-gateway thing is not available/implemented yet, so maybe some of these questions can be answered by reading the code)

What's attribution like? I'd like to see commits of this form:

Author: Qais Patankar <[email protected]>
Committer: Better Informatics
 <42103905+better-informatics[bot]@users.noreply.github.com>
Better Informatics
 <42103905+better-informatics[bot]@users.noreply.github.com>
Subject: <some user provided change reason. or if they don't give a reason, some autogenerated reason like "Update inf3/ct.md">

Where the real name and email address is derived from their GitHub account (it would show the same details as if they did it in the github web interface).

There's some obvious complexity and barrier issues here (requiring the user to log in, but not requiring them to understand the rest of the GitHub interface). We can chat about that.

In general what I'd like to (/ would not like to) see are:

1. (+) want to be able to differentiate machine commits from each other (which is why we should set the commit Author field)

2. (+) using (1) above, people should also get commit credits on GitHub (see [2*])

3. (+) should be able to differentiate commits done through the interface, from commits made manually by the user via normal git. (hence Committer != Author)
   (this also makes semantic sense. the machine made the commit on behalf of the user, and pushed via the BI app)

4. (+) want to see good commit messages. so it'd be good if they could provide a reason when they save a change.

   (obviously we can't expect people to provide good messages in the input field. but I would hate for it to be impossible.)

   (-) basically don't want to see machine commits with meaningless commit messages

[2*] For people to get commit credits, they'd have to be added to the org and team: https://github.com/orgs/compsoc-edinburgh/teams/better-informatics/members. That process is manual at the moment. WHenever I see someone make a PR I just invite them.

Editing the front matter is explicitly not supported, because that should be a fairly rare case, and should be done by a maintainer, ideally.

maintainer: are you thinking of limiting the ability to edit metadata / edit the repo directly? Or reduce the chance of a fuckup by not adding that ability to the web interface? (Just want to understand your thoughts here)

Proof of concept is now up at bi.comp-soc.com

(I have to ask) is the URL just for the proof of concept? 🙂

---

All in all! Well done. Looks great so far :)

Some edge cases to not forget about, in case it might be affected by online editing:

• non-course sections ("General" on each year page. And the entire homepage sections.)
• Non-Inf courses
• Local development (right now local development is quite shitty. New stuff should maybe try to keep this in mind?)

I'm really excited to see what's next and, if you decide to go ahead with this iteration, see this get deployed.

Once we have online editing / this new framework down, we're no longer restricted to a static website, and actually have a base to work off. There's tons of possibilities (... such as tighter drive integration for past papers etc.)

I'm hyped! 👌

[qaisjp]

P.S. in case you didn't know, we have this app set up here for API operations: https://github.com/organizations/compsoc-edinburgh/settings/apps/better-informatics

[pkage]

Heya! Somehow i completely missed this in my GH notifications.

Lemme start off by saying I really, really like this. Editing seems super smooth, and I dig how you can immediately get a WYSIWYG preview of your edits. I think that in-place editing of BI would lower the barrier for entry for maintaining BI—a lot of people aren't comfortable with Git or web dev (or both).

With that said, I have some concerns about maintainability. This change adds a heck of a lot of dependencies onto the project, namely a rewritten templater (Svelte), a cloudflare worker/GH app to build the page bundle, and netlify for serving that static bundle. While these are all super cool, I think that this is fairly fragile. What happens if Cloudflare changes their pricing? Or if GH apps disappears altogether? Or netlify starts charging? Plus, with all these cloud services backing up the service local development is difficult to do.

Ultimately, BI isn't really in a position to take on new technical debt. Our goal should be to ensure that BI is available as a resource to future generations of students, and to that end the website should have the minimum of moving parts possible—and those moving parts should be as boring as possible.

With that in mind, I think the edit-on-Github flow works pretty well. Most students have a github account, each section is individually editable, and merge conflicts (should they happen) are very clearly marked. GH pages for hosting—while it is aggressively cached—still updates within 5-10 minutes of the underlying material changing. This is plenty fast enough for what is ultimately a read-heavy workflow. GH pages isn't going anywhere, Jekyll isn't going anywhere, why not build off those?

Alternatively, if we do need a BI interface with attribution+proper merging+good history support+WYSIWYG editor, should we think about using an off-the-shelf wiki solution? Something like Gollum seems like it would fit our needs without fully re-inventing the wheel.

---

I really really don't want this to sound like an attack—I absolutely love bi.comp-soc.com. It's a really cool project with some very clever code behind it. I'm just concerned about the maintainability going forward. If there's a way to do in-place editing while also limiting the complexity, I'm all for it.

[himmelmaus]

So I also really like this, but at the same time I agree with kage's concerns to an extent. I think this is a good proof of concept but it could definitely do with being streamlined. One more thing which I would be concerned about is vandalism, unless I'm missing something there's no login or way of tracking who has done what in this implementation - and I think we would need some sort of login (probably with cosign?) and way of blacklisting people to ensure we don't end up with vandalism taking place.

I know this is a quick PoC though, so I'm not saying this as a criticism - more something we should heavily consider if we do go ahead with this idea. One other thing is that I like how bespoke BI is and wouldn't want to move to a completely out the box wiki service because I feel we would lose some of the "homeliness" for want of a better word, on the frontend at least.

[penalosa]

Thank you all for the feedback, I very much take it in the spirit it was intended and not as an attack. To go through things one by one:

How does that work? Just it just serve from the gh-pages branch (from which repo?)

There's a GitHub Action which compiles the site into the gh-pages branch, which Netlify then serves. I think something important to note here (and feeding into @pkage's concerns about dependencies on cloud services), is that Netlify doesn't do any building, and this is fully compatible with how GitHub Pages hosts things (we could even throw the build output into an S3 bucket). The reason it's hosted by Netlify is purely because of GitHub's aggressive caching. While there is admittedly a dependency on GitHub Actions, that could (if need be) be replaced with a script running on a server somewhere (in a similar manner to current BI) that responded to webhooks.

WYSIWYG

This would be a bit of a tricky one, especially in terms of preserving markdown compatibility. I haven't specifically looked into ckeditor, but I know QuillJS has fairly good support for putting strict constraints on the content, which might allow us to translate it back to markdown.

Is this the frontend connecting to the backend here? https://github.com/compsoc-edinburgh/bi-engine/blob/master/save-section/index.js

It is indeed. It connects through a Git Gateway instance setup with the correct repo and my GitHub account, but that's just for right now—it could absolutely use a GitHub app in future. All it does is call the needed GitHub API endpoints to make a commit with a file modification.

What's attribution like?

This was a quick POC and so the commit messages are admittedly terrible 😂. That's definitely something to fix. Requiring a commit message before saving is a good thought, it should help with keeping track of history. The bigger auth question though is definitely something to be discussed. Right now (as you mentioned @findoslice) there's no authentication or identity checking, and I agree that could lead to vandalism. I have a hope that it wouldn't, and that our fellow students are better than that, but realism must prevail. I think in general I'm opposed to requiring GitHub accounts, because although a lot of students do have accounts not all of them do, and they probably wouldn't create one just to add some notes. I'd agree with having an option between GitHub and CoSign though—seems like the best of both worlds, with students either able to get commit credits on GitHub or have a really low barrier to entry.

maintainer: are you thinking of limiting the ability to edit metadata / edit the repo directly? Or reduce the chance of a fuckup by not adding that ability to the web interface? (Just want to understand your thoughts here)

Not necessarily thinking of limiting it, but in general I don't think being able to edit the metadata for a section should have a lowered barrier to entry, as it shouldn't happen very often, and could have unintended side effects.

I have some concerns about maintainability

You're not wrong. This adds a ton of complexity, and it may ultimately not be worth it. I do think though that improving the editing experience is worth a bit of extra complexity, although admittedly not necessarily as much as added here. Some suggestions to help with that:

• Netlify isn't really needed: It only serves the static content, and since the frontend includes a preview of your changes until the underlying site rebuilds and purges the cache, caching time isn't really an issue, and this could go back to GitHub Pages.
• Building of the site happens in a GitHub Action: Honestly, I'm not sure this is a huge concern. Actions probably aren't going away anytime soon (and if they do, things like GitLab have pretty comparable products), are free for public repos, and seem to me to be more maintainable than a rebuild script sitting on a server responding to a web hook (although we could absolutely do that as an alternative).
• Cloudflare workers: This is a bit more of an iffy one—they're a fairly new platform, and could well get rid of their free tier at some point (or change how they work). None of the logic here is that specific to CF Workers, and so this could be moved to a dockerised script running in CompSoc's DO account (and probably should be).

There's a lot here, so I've probably missed a point or two (if so, do bring them up again)

[qaisjp]

• I agree with @pkage & @findoslice's comments. Although, I agree with you @penalosa about GitHub Actions, I don't think it is going away anytime soon.

• I think some of the core points here about <GitHub Pages, Netlify, Cloudflare> revolve around moving away from a VM hosted on Tardis. Is there a particular reason y'all wanna do that?

  I get with the CompSoc server you don't want to maintain the hardware, but I've found that BI has lived pretty happily on Tardis.

  Yeah, the bi.tardis.ed.ac.uk VM has poor documentation, but that's because it's just been me and @sleepyfoxen maintaining the VM. And nobody has really said "hey I wanna do X, can you give me access and show me how to get started?".

  I've been adding more and more docs to https://github.com/compsoc-edinburgh/bi-app. If more people take the initiative to try and get involved with the more sysadminny sort of things, I can identify the pain points, and write tutorials & documentation.

• Vandalism: here's some extra history. Better Informatics used to be a website with a password. Hover over the dromy (camel) on this page: web.archive.org/(2013-06-22)/betterinformatics.com/login.php. Then, after you logged in, there was an edit button for each year, which would show a textbox to input HTML.

  Now, I'm not suggesting we block the entire site behind a password, but if you felt abandoning CoSign or GitHub was an option, you could block editing behind a password (and maybe captcha).

• Ultimately, BI isn't really in a position to take on new technical debt. — @pkage

  I strongly agree. I also think we should stick to languages and technologies that are the most accessible to students (and maybe even DICE machine friendly).

  Our goal should be to ensure that BI is available as a resource to future generations of students, and to that end the website should have the minimum of moving parts possible—and those moving parts should be as boring as possible. — @pkage

  I strongly agree with this too.

  Over the years mapp has had a few small features added here and there with the aim to keep the main site lightweight, which has resulted in a couple of confusing moving parts. More comments on that below:

  I think there's a balance between a monolithic application & "keeping the main site lightweight".

  When we keep the main site lightweight, there's a lot of confusing moving parts. That what we have right now. A lot of things can go wrong, but if something breaks, it doesn't affect anything else. (And it's been incredibly easy to failover to GitHub Pages, which I have done once or twice in the past.)

  When we have a monolithic application, it's much easier to maintain. But if something breaks, everything goes down. The web.archive.org link above links to a Better Informatics that was monolithic, but thankfully only editing was broken, and thankfully the login & regular viewing still worked fine.

  I don't think moving towards a monolithic application is a bad idea. (Just as long as the data is not locked away in a database. Keeping this repo as a data source helps make it consumable for future years. If a monolithic application dies, the data won't decay in a database somewhere.)

• Local development

  Additional info about CoSign: developing things that depend on CoSign is a pain in the arse, so you either need to:

  1. test in production, or
  2. have a "staging" environment, in this case of mapp (this link is supposed to 502), or
  3. add "dev mode authentication", which doesn't use CoSign SSO, or
  4. host a local version of CoSign - http://weblogin.org/

  (If I rewrote mapp today, I would go for "dev mode authentication".)