DOCS: TOC builder (for functions and providers) #2197

systemcrash · 2023-03-17T17:00:34Z

first attempt to automate TOC creation.

This is a draft so there's probably a few misses in here to sort through after some test doc generations.

Summary
Automatically generate the TOC in SUMMARY.md and fill it with the hierarchical doc structure found in the documentation folder. Now we don't need to manually update it or its links. Folder structure depth is mirrored in the TOC.

It's possible to supply file/folder name exceptions at various points.

Checks are simple and can possibly be defeated.

Due to the (renames and) changes for this PR, it's not advisable to 'merge master' into this branch.... yet.

systemcrash · 2023-03-17T17:05:10Z

OMG. So much structure. So many doc. Everything still appears intact on the doc builds. Could do with some polish for filenames.

systemcrash · 2023-03-17T19:24:07Z

I may invest some time to program a scour mechanism that goes through all markdown and updates all the links if a file has moved location or got renamed. But that can come later.

build/generate/docuTOC.go

systemcrash · 2023-03-17T20:02:43Z

That would work.

cafferata · 2023-03-17T20:04:31Z

Thanks @systemcrash! Another step in the right direction! I myself have had similar ideas see #1998. I just don't know if I would make the same choice to generate the full SUMMARY.md myself. With this you lose the ability to show better menu names than in the new situation.

Some examples:

-CI/CD example for GitLab
+Ci-Cd-Gitlab

-Why CNAME/MX/NS targets require a "dot"
+Why Targets Require The Final Dot

I've always thought of two (or more) GitBook menu structures:

I'm curious about your thoughts! @systemcrash @tlimoncelli

systemcrash · 2023-03-18T16:08:29Z

OK - I'm satisfied with the results now.

Thanks @systemcrash! Another step in the right direction! I myself have had similar ideas see #1998. I just don't know if I would make the same choice to generate the full SUMMARY.md myself. With this you lose the ability to show better menu names than in the new situation.

Some examples:
-CI/CD example for GitLab
+Ci-Cd-Gitlab
-Why CNAME/MX/NS targets require a "dot"
+Why Targets Require The Final Dot

Just a question of renaming the file (as closely) as possible. Or modifying the code to pick out the first # header from each md and updating every md to have one, if perfection is required. But perfection is often the enemy of good.

I've always thought of two (or more) GitBook menu structures:
1. [The providers](https://github.com/StackExchange/dnscontrol/blob/f912b15/documentation/SUMMARY.md?plain=1#L89-L129)

2. [The language reference](https://github.com/StackExchange/dnscontrol/blob/f912b15/documentation/SUMMARY.md?plain=1#L15-L83)
I'm curious about your thoughts! @systemcrash @tlimoncelli

Do you think they're sufficiently mirrored in the current test results? Take a look and let me know @cafferata

systemcrash · 2023-03-18T16:27:02Z

In tandem with this PR, I recommend that (a few) providers might also be renamed (and their variables too?) to match the docs. Things like Route 53 are findable by search, although this one is actually called "Amazon Route 53".

( PR has overview of which provider md files got renamed )

cafferata · 2023-03-18T18:14:15Z

@systemcrash a number of feedback points:

The absolutes file paths are not supported by GitBook. This causes the URLs to not work at this time.
By redesigning folders/summary, this pull request not only has a generation process been built, but also a new look and feel. As far as I'm concerned, we take it step by step so that there is clarity about exactly what has changed and for what reason. Perhaps @tlimoncelli thinks differently about this? As far as I'm concerned, this state of pull request is not ready to merge.
~~See also DOCS: TOC builder (for functions and providers) #2197 (comment)~~

Before	After

systemcrash · 2023-03-18T22:06:01Z

Yes - thanks for the feedback. I made a few minor adjustments to retain the original structure as closely as possible. Seems to have worked nicely. The only part that won't align is the provider names - the provider .md files and the providers need renaming to synchronize or, another solution.

When naming freely the provider md files, it looks like you want:
https://docs.dnscontrol.org/~/revisions/284FcG65SMBYAdV2iV3g/providers/providers

If there is some structuring you don't like, one just has to rearrange the files/folders (bearing in mind to adjust any hyperlinks.)

I think this is where we gain by "good code documents itself" - by renaming the providers so everything looks as you want it.

systemcrash · 2023-03-22T23:10:38Z

Hi @cafferata - are there any other changes you'd like to see for the long term?Or is it now just a question of modifying the processes on the docs server?

cafferata · 2023-03-23T16:13:26Z

I'm especially curious if @tlimoncelli agrees with the structure change.

The new menu structure shows the technical details/directory names.

-Getting Started
+01 Getting Started

-Language Reference
+02 Language Reference

Determining the order of menu items is no longer possible.

Example Getting Started menu items

-Overview
-Examples
-Migrating zones to DNSControl
-TypeScript autocomplete and type checking
+Examples
+Getting Started
+Migrating Zones To Dnscontrol
+TypeScript Autocomplete And Type Checking

Example Language Reference

-Top Level Functions
-Domain Modifiers
-Record Modifiers
+Domain Modifier Functions
+Record Modifier Functions
+Top Level Functions

The chapters within the menu structure

-Service Providers
-Commands
-Advanced features
-Developer info
-Release
+Advanced Features
+Commands
+Developer Info
+Providers
+Release

External links
Randomly adding external links, such as the GitHub releases, is no longer possible.
Provider names that change
```
-Microsoft DNS Server on Microsoft Windows Server
+Msdns
```
- https://docs.dnscontrol.org/~/revisions/zK7MqMAiAJbsWHSHv29Q/providers/providers/msdns
- https://docs.dnscontrol.org/service-providers/providers/msdns
Absolutes file paths DOCS: TOC builder (for functions and providers) #2197 (comment)
This causes the URLs to link to the GitHub markdown view instead of GitBook documentation URLs.
Question, how are the renames ensured that this goes well? How has this been tested?
1. Example Introduction to DNSControl > Quick start tutorial
  - https://github.com/StackExchange/dnscontrol/pull/2197/files#diff-8a992fc8a61d84dc7e49e3b126235f05599212988c01f9b712517995b9a59531R7
  - https://docs.dnscontrol.org/~/revisions/zK7MqMAiAJbsWHSHv29Q/#:~:text=quick%20start%20tutorial
2. Example Introduction to DNSControl > migrate
  - https://github.com/StackExchange/dnscontrol/pull/2197/files#diff-8a992fc8a61d84dc7e49e3b126235f05599212988c01f9b712517995b9a59531R7
  - https://docs.dnscontrol.org/~/revisions/zK7MqMAiAJbsWHSHv29Q/#:~:text=migrate
3. Example Introduction to DNSControl > language spec
  - https://github.com/StackExchange/dnscontrol/pull/2197/files#diff-8a992fc8a61d84dc7e49e3b126235f05599212988c01f9b712517995b9a59531R7
  - https://docs.dnscontrol.org/~/revisions/zK7MqMAiAJbsWHSHv29Q/#:~:text=language%20spec
4. Example Providers > Provider table > Domain modifier deeplink
  - https://github.com/StackExchange/dnscontrol/pull/2197/files#diff-28e66bede643a7156cf9c19950731e1a3a9ab66becf1a063f7705076bfc4dbdaR78
  - https://docs.dnscontrol.org/~/revisions/zK7MqMAiAJbsWHSHv29Q/providers/providers
5. Example Record Modifier Functions > Service Provider Specific > Amazon Route 53 > R53_ZONE
  - https://github.com/StackExchange/dnscontrol/pull/2197/files#diff-64fa1fd972040542557ac120c90268ff7e8e6fa6407826588f2cf9a82fdbdf7aR15
  - https://docs.dnscontrol.org/~/revisions/zK7MqMAiAJbsWHSHv29Q/02-language-reference/record-modifier-functions/service-provider-specific/amazon-route-53/r53_zone

systemcrash · 2023-03-23T16:51:13Z

I'm especially curious if @tlimoncelli agrees with the structure change.
1. The new menu structure shows the technical details/directory names.
   ```diff
   -Getting Started
   +01 Getting Started
   ```

I don't see the problem with technical details. If anything, it makes it easier for new-comers to find and contribute.

I did not find a convenient way of forcing something to appear first, when everything is sorted alphanumerically, except perhaps naming entries G... instead of g... and sorting alphabetically (A-Z first, then a-z). That has its own problems, eg entries that begin DNS.... This whole project is about DNS...

What is the documentation trying to be? Reference? Howto? User guide (examples)? Reference should be alphabetically sorted (providers, functions). Howtos can land in a different folder and get a different TOC...? But we're trying to avoid all of this.

   ```diff
   -Language Reference
   +02 Language Reference
   ```

2. Determining the order of menu items is no longer possible.
   
   1. **Example Getting Started menu items**
      ```diff
      -Overview
      -Examples
      -Migrating zones to DNSControl
      -TypeScript autocomplete and type checking
      +Examples
      +Getting Started
      +Migrating Zones To Dnscontrol
      +TypeScript Autocomplete And Type Checking
      ```
   2. **Example Language Reference**
      ```diff
      -Top Level Functions
      -Domain Modifiers
      -Record Modifiers
      +Domain Modifier Functions
      +Record Modifier Functions
      +Top Level Functions
      ```
   3. **The chapters within the menu structure**
      ```diff
      -Service Providers
      -Commands
      -Advanced features
      -Developer info
      -Release
      +Advanced Features
      +Commands
      +Developer Info
      +Providers
      +Release
      ```

Do we need to? 'Everything else' in the project is sorted alphabetically, which makes it easier to locate entries (reference).

3. External links
   Randomly adding external links, such as the [GitHub releases](https://github.com/StackExchange/dnscontrol/releases/latest), is no longer possible.

Such entries need to land outside of the tags, true. Other 'decorative' static entries can land somewhere in the middle, but that means more tags and more code complexity. I would rather code be easy to understand.

'Randomly' adding content goes into an existing file or makes a new file somewhere, which gets checked in anyway. Don't see the problem.

4. Provider names that change
   ```diff
   -Microsoft DNS Server on Microsoft Windows Server
   +Msdns
   ```

I made an earlier suggestion that we rename provider entries, to align with the real world. Only drawback is that users might have to (oh no) change a text string in their .js files.

e.g. hedns -> hurricane_electric_dns (underscores replaced to spaces). Frankly I find real world naming much easier. Putting some of the short forms into google to find who they were was difficult for some, and uncertain for others.

   * https://docs.dnscontrol.org/~/revisions/zK7MqMAiAJbsWHSHv29Q/providers/providers/msdns
   * https://docs.dnscontrol.org/service-providers/providers/msdns

5. Absolutes file paths [DOCS: TOC builder #2197 (comment)](https://github.com/StackExchange/dnscontrol/pull/2197#issuecomment-1474936257)
   This causes the URLs to link to the GitHub markdown view instead of GitBook documentation URLs.

So... how does one fix this, if gitbook is the solution? Relative but with some path depth?

   **Question**, how are the renames ensured that this goes well? How has this been tested?
   
   1. **Example `Introduction to DNSControl` > `Quick start tutorial`**

The renames were made to follow how they are manually named in the TOC. Just rename the source file. 1:1 naming. Everything else was just link decoration. Now it's automatic. File == page == link.

      * https://github.com/StackExchange/dnscontrol/pull/2197/files#diff-8a992fc8a61d84dc7e49e3b126235f05599212988c01f9b712517995b9a59531R7
      * https://docs.dnscontrol.org/~/revisions/zK7MqMAiAJbsWHSHv29Q/#:~:text=quick%20start%20tutorial
   2. **Example `Introduction to DNSControl` > `migrate`**
      
      * https://github.com/StackExchange/dnscontrol/pull/2197/files#diff-8a992fc8a61d84dc7e49e3b126235f05599212988c01f9b712517995b9a59531R7
      * https://docs.dnscontrol.org/~/revisions/zK7MqMAiAJbsWHSHv29Q/#:~:text=migrate
   3. **Example `Introduction to DNSControl` > `language spec`**
      
      * https://github.com/StackExchange/dnscontrol/pull/2197/files#diff-8a992fc8a61d84dc7e49e3b126235f05599212988c01f9b712517995b9a59531R7
      * https://docs.dnscontrol.org/~/revisions/zK7MqMAiAJbsWHSHv29Q/#:~:text=language%20spec
   4. **Example `Providers` > `Provider table` > Domain modifier deeplink**

This one I'm not sure how to resolve - I don't know what happens server side. ¯_(ツ)_/¯
But I'll stick with something that works.

      * https://github.com/StackExchange/dnscontrol/pull/2197/files#diff-28e66bede643a7156cf9c19950731e1a3a9ab66becf1a063f7705076bfc4dbdaR78
      * https://docs.dnscontrol.org/~/revisions/zK7MqMAiAJbsWHSHv29Q/providers/providers
   5. **Example `Record Modifier Functions` > `Service Provider Specific` > `Amazon Route 53` > `R53_ZONE`**

Same content.

      * https://github.com/StackExchange/dnscontrol/pull/2197/files#diff-64fa1fd972040542557ac120c90268ff7e8e6fa6407826588f2cf9a82fdbdf7aR15
      * https://docs.dnscontrol.org/~/revisions/zK7MqMAiAJbsWHSHv29Q/02-language-reference/record-modifier-functions/service-provider-specific/amazon-route-53/r53_zone

Documentation search precludes any of this from being a problem. Know what you want to find? ⌘K.

tlimoncelli · 2023-03-23T18:41:48Z

I'm especially curious if @tlimoncelli agrees with the structure change.
The new menu structure shows the technical details/directory names.
-Getting Started
+01 Getting Started
-Language Reference
+02 Language Reference

My concern here is that if we add a section a lot of links will break. I would prefer a system where the name the user sees is not the directory name. This will also permit us to use characters that can't be in a directory name, such as /.

Each file has a preamble (for example documentation/functions/domain/FRAME.md). Could this be part of the solution?

Determining the order of menu items is no longer possible.

That's also a problem. It is important that articles are listed in an order that improves usability/readability. For example, the more broad/introductory entries should be at the top of the list.

(Fun fact: When you are at a restaurant for the first time and don't know what to order, the first 1-2 items in any section of the menu are the "safe" items that a new and non-adventurous eater should pick. Likewise, Apple's style-guide says that the items in a drop-down menu should be in the order a typical user will learn them. That's why Open comes before Save, and the last few items in the drop-down menu are the obscure features that most people won't use)

External links
Randomly adding external links, such as the GitHub releases, is no longer possible.

We only have 1 such link. No biggie. We could delete it or have a doc that tells people to go to the link.

Provider names that change

-Microsoft DNS Server on Microsoft Windows Server
+Msdns

This is fine. Though we probably want to shorten some of the full descriptions.

cafferata · 2023-03-23T18:45:42Z

Hi @systemcrash, your response feels personal. @tlimoncelli has asked me to provide feedback/view of this pull request. So I've listed the changes that aren't immediately obvious from the GitHub pull request description (or the big git diff). In addition, I explicitly asked how @tlimoncelli looks at these feedback points. Now it feels like we have to discuss each other's comments and that's not up to me.

Would I personally have done it differently? Yes, I would have chosen to narrow the scope down to what you call 'reference' (language reference and providers).

Am I grateful that you have taken up this work? Absolute!

systemcrash · 2023-03-23T21:21:12Z

Hi @systemcrash, your response feels personal. @tlimoncelli has asked me to provide feedback/view of this pull request. So I've listed the changes that aren't immediately obvious from the GitHub pull request description (or the big git diff). In addition, I explicitly asked how @tlimoncelli looks at these feedback points. Now it feels like we have to discuss each other's comments and that's not up to me.

Would I personally have done it differently? Yes, I would have chosen to narrow the scope down to what you call 'reference' (language reference and providers).

Am I grateful that you have taken up this work? Absolute!

Better we find a long-term solution than just mashing this in there, since I understand that you put in some work server-side to make this work out. And yes, it's personal: it's my opinion based on this work.

How about naming files with a number prefix for some kind of ordering, but the automation trims the number prefix away?

The only other way I can think of is embedding the title you want in the first line(s) of the .md files (in certain folders) and using those. More complex, but it seems to strike a balance. just have to go through all files and ensure the first line always adheres to this standard. ( Similar to the preamble @tlimoncelli mentions, but not for alphabetically sorted reference stuff )

tlimoncelli · 2023-03-24T18:48:12Z

@systemcrash I appreciate all the effort you're putting into this. Good, readable, documentation is probably the most important way we bring new people into the project.

I like all the ideas so far. Here are my thoughts about them:

Filename prefixes: Use 3-digit numbers with big gaps between them (100_foo, 200_bar, etc) so that new files can fit between them. (Did you program in BASIC on an 8-bit computer?) The downside is that it may be confusing to not see the numbers in the URL.
File preamble: Add a preamble to the individual files: Hugo does this. The file has a preamble where you can set variables. For example add weight = 5 to a file's preamble and it will be listed before a file with weight = 6. Files without preambles are listed last.
A "toc file". That is... if a directory includes a file called toc.yaml, that file could list the filenames in the order they are to appear (files not listed can go at the end).

Thoughts?

Tom

systemcrash · 2023-03-24T20:29:44Z

OK, updates:

The TOC auto-creation is reduced in scope now to only ref material, and providers.
Each section has its own  (replacement function more generic now).
I added a link rectifier which replaces links to other .md files with a relative link.
- This operates on files within specified folders - see docuLocalLinkRectifier.go and generate.go

As a result, all links work. TOC is auto-generated. Everybody is happy (I think).

(I may have missed a link or two somewhere).

renames/moves:
providers -> service_providers
functions (etc) -> language_reference

We can increase scope/complexity of auto-generation later with these ideas if we are satisfied with how this currently works

systemcrash · 2023-04-03T22:56:11Z

Shall we use this automation?

(it has more error checks)

systemcrash force-pushed the toc_builder branch from c642e49 to 4704075 Compare March 17, 2023 17:01

systemcrash changed the title ~~DOCS: Toc builder~~ DOCS: TOC builder Mar 17, 2023

systemcrash force-pushed the toc_builder branch 2 times, most recently from 1c05618 to 5491262 Compare March 17, 2023 17:49

systemcrash mentioned this pull request Mar 17, 2023

DOCS: Fixed LOC_BUILDER_STR filename #2199

Merged

tlimoncelli reviewed Mar 17, 2023

View reviewed changes

build/generate/docuTOC.go Outdated Show resolved Hide resolved

systemcrash force-pushed the toc_builder branch 6 times, most recently from 8e269b5 to 0f106ec Compare March 18, 2023 16:02

systemcrash force-pushed the toc_builder branch from 0f106ec to c6cceb5 Compare March 18, 2023 16:22

systemcrash marked this pull request as ready for review March 18, 2023 16:24

systemcrash force-pushed the toc_builder branch 2 times, most recently from a19f6de to 6e85342 Compare March 18, 2023 16:42

systemcrash force-pushed the toc_builder branch 3 times, most recently from 9e15761 to 7eea9f1 Compare March 18, 2023 22:00

systemcrash force-pushed the toc_builder branch 3 times, most recently from b28b1fd to 35f7f03 Compare March 18, 2023 22:41

systemcrash force-pushed the toc_builder branch from a233784 to df31642 Compare March 23, 2023 00:00

systemcrash marked this pull request as ready for review March 23, 2023 00:00

systemcrash marked this pull request as draft March 24, 2023 01:45

systemcrash force-pushed the toc_builder branch 2 times, most recently from b4fd6bb to aa8b620 Compare March 24, 2023 20:26

systemcrash marked this pull request as ready for review March 24, 2023 20:29

systemcrash force-pushed the toc_builder branch from aa8b620 to 074c6e2 Compare March 24, 2023 20:32

systemcrash changed the title ~~DOCS: TOC builder~~ DOCS: TOC builder (for functions and providers) Mar 24, 2023

systemcrash mentioned this pull request Mar 25, 2023

request: rename all providers to match real world #2233

Closed

systemcrash force-pushed the toc_builder branch 2 times, most recently from 26d042e to 46811d4 Compare March 26, 2023 00:34

systemcrash added 9 commits April 4, 2023 00:57

Code to auto build TOC entries based on folder structure

757a1fa

Change featureMatrix to use the new replace function

0238b08

(it has more error checks)

featureMatrix shall use new service_providers path for links

2e40fc4

Refresh

3af6441

Autofix Markdown Links for files in a folder

22ba828

Amend links

ce61da5

Amend links (in all .md files)

b310ada

Amend adding new rtypes procedure.

d8c3904

Refresh 2

11d6dc2

systemcrash force-pushed the toc_builder branch from 46811d4 to 11d6dc2 Compare April 3, 2023 23:01

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

DOCS: TOC builder (for functions and providers) #2197

DOCS: TOC builder (for functions and providers) #2197

systemcrash commented Mar 17, 2023

systemcrash commented Mar 17, 2023

systemcrash commented Mar 17, 2023

systemcrash commented Mar 17, 2023 via email

cafferata commented Mar 17, 2023

systemcrash commented Mar 18, 2023

systemcrash commented Mar 18, 2023 •

edited

Loading

cafferata commented Mar 18, 2023 •

edited

Loading

systemcrash commented Mar 18, 2023 •

edited

Loading

systemcrash commented Mar 22, 2023

cafferata commented Mar 23, 2023

systemcrash commented Mar 23, 2023

tlimoncelli commented Mar 23, 2023

cafferata commented Mar 23, 2023

systemcrash commented Mar 23, 2023 •

edited

Loading

tlimoncelli commented Mar 24, 2023

systemcrash commented Mar 24, 2023 •

edited

Loading

systemcrash commented Apr 3, 2023

DOCS: TOC builder (for functions and providers) #2197

Are you sure you want to change the base?

DOCS: TOC builder (for functions and providers) #2197

Conversation

systemcrash commented Mar 17, 2023

systemcrash commented Mar 17, 2023

systemcrash commented Mar 17, 2023

systemcrash commented Mar 17, 2023 via email

cafferata commented Mar 17, 2023

systemcrash commented Mar 18, 2023

systemcrash commented Mar 18, 2023 • edited Loading

cafferata commented Mar 18, 2023 • edited Loading

systemcrash commented Mar 18, 2023 • edited Loading

systemcrash commented Mar 22, 2023

cafferata commented Mar 23, 2023

systemcrash commented Mar 23, 2023

tlimoncelli commented Mar 23, 2023

cafferata commented Mar 23, 2023

systemcrash commented Mar 23, 2023 • edited Loading

tlimoncelli commented Mar 24, 2023

systemcrash commented Mar 24, 2023 • edited Loading

systemcrash commented Apr 3, 2023

systemcrash commented Mar 18, 2023 •

edited

Loading

cafferata commented Mar 18, 2023 •

edited

Loading

systemcrash commented Mar 18, 2023 •

edited

Loading

systemcrash commented Mar 23, 2023 •

edited

Loading

systemcrash commented Mar 24, 2023 •

edited

Loading