tools/read

tools/read/README.md · 17674 B · source sha256 077ff5a106d1

read_gen.py builds the site. The site is the repository root: it copies the repository as authored, except vercel.json, _site/, hidden files, and hidden and symlinked directories, and adds the pages it generates under read/: one reading page per declared document, pattern contract and record, and a documents index.

The site root keeps the authored index.html, which declared.json names as the home page, and every generated page's breadcrumb links it.

The Markdown stays the only source. The generated pages are build output: they are written to the directory the build is given and never committed. Editing a Markdown file is enough to change its page at the next build, with three exceptions, each of which needs declared.json changed in the same commit: a new Markdown file anywhere in the repository needs a declared place; a new blockquote needs a declared role; and changing the opening words of a declared passage or block needs its declaration changed to match. Until then the build fails and nothing new is deployed.

The pages are static files. Each loads same-site stylesheets, fonts, icons and two scripts, surface-shell.js and surface-document-overflow.js; the shell script builds the responsive navigation panel from the page's own markup. No content is generated or fetched at runtime, and the site runs no server-side code.

GitHub stays the source of record. Every reading page links its file on GitHub at main and its raw file. Every file the repository serves today is still served at its own path, and the style guide, the pattern previews and the keys are unchanged apart from the composition below.

on this page

Provenance

read_gen.py, vercel-build.sh and requirements.txt are adapted from apexSolarKiss/method-ASK at 5e3a332164b2973d9eea2337ddeed825ae56ad4d (tools/read/). The renderer, the gates and the page markup are method-ASK's. What differed by repository is declared in declared.json instead of written into the code: the repository's name and index copy, the stylesheet, script and wordmark paths, the site directory, the home page, the front doors, the document collections and their output names, default holds, and directory exclusions (site_exclude entries ending in /), inside which the build's output and report may land. Five behaviors were added: a link to a directory inside the site with no index.html of its own, to a Markdown file in the site that is not rendered, or to a hidden file opens on GitHub rather than as a raw file, a missing listing or a file the site does not serve; hidden files are not copied into the site; an authored home page is link-checked like a generated one; a word too long for its line breaks rather than widen the page at narrow widths; and the build's output may sit inside the site only within a directory the declaration excludes, matched case-exactly even before that directory exists, where method-ASK's refused any output inside the site. Two navigation details also differ from that commit, and method-ASK's copy changes the same way: a generated page's mark links the home page rather than the documents index, and a source view's navigation panel lists the documents among its utilities rather than under the diagrams. Each generated page's link preview (see The declaration) was added later, to this copy and to method-ASK's alike. requirements.txt is method-ASK's, unchanged. This repository owns its adapted copy of read_gen.py, and declared.json supplies the repository-specific configuration. The copy reads declaration keys a site may not use — home_page, held_under, collections or a repository-root site — and a key absent from declared.json has no effect.

The front doors

Three authored pages are composed on their way into the site, and no other authored file changes.

  • patterns/index.html links each pattern's README and each semantic primitive's Markdown. Opened from a clone it opens those raw files; in the composed site each of those links opens the file's reading page. A query or #fragment on the link is kept.
  • index.html gains a documents route, and documents joins its footer and its navigation panel. Each entrance sits in a <template data-read-build> element, whose content is inert, so a clone shows no dead link; the build removes the template tags, each with its line where it stands alone on one, so the entrances render.
  • patterns/index.html and preview/styleguide.html gain the same documents entrance in their footers and navigation panels, in the same build-only elements.

Apart from the mapped href values and those template tags, each composed copy is the authored bytes.

Deployment

vercel.json runs the build. Vercel runs sh tools/read/vercel-build.sh at the repository root and serves _site. The script:

  1. picks a Python 3.10 or later on the build machine, or stops;
  2. installs the exact renderer pin into a temporary virtual environment, by hash, from wheels only;
  3. runs read_gen.py build --out _site at the repository root.

A failed gate fails the build, so a broken document is never deployed. vercel.json also carries the site's headers, redirects and rewrites; the build changes none of them, and /styleguide/ and /patterns/ resolve as before. It also redirects /read to /read/, so the documents index, served from a directory, keeps its relative links.

Setup

For local builds you need Python 3.10 or later, with the exact renderer pin installed in a virtual environment outside the repository:

python3 -m venv <venv>
<venv>/bin/python -m pip install --require-hashes --only-binary=:all: -r tools/read/requirements.txt

Every mode checks the installed versions first. It exits 2, running nothing, unless they are exactly markdown-it-py 4.2.0, mdit-py-plugins 0.6.1 and mdurl 0.1.2.

Use

From the repository root:

<venv>/bin/python tools/read/read_gen.py build --out _site
<venv>/bin/python tools/read/read_gen.py check
<venv>/bin/python tools/read/read_gen.py links --site _site
  • build writes the composed site to --out. That directory must not exist yet, or must be empty. It may not be the repository, sit inside the repository or .git (except _site/, which the declaration excludes from the site), or contain the repository. These are judged by where --out physically lands: every symlink in the path, and any .. after one, is followed first, and when the check fails nothing is staged. Serve _site/ and open /read/index.html to read the result. --report <file> writes the pages, link classes, headings and the front-door mapping after a successful build. It may not land inside the repository or .git (except _site/, which the declaration excludes from the site), and it may not be --out, lie inside it or contain it, judged the same way. That holds whether --out exists yet or not: when it does not, the build creates its missing directories for the comparison, so the volume itself decides which spellings name the same directory, and removes them again before anything is staged.
  • check runs the same build into a temporary directory, prints the digest and removes the directory.
  • links runs the link check alone over a composed site.

Each successful build prints a digest of the composed tree. Identical inputs give identical digests.

Gates

The build renders every page in memory first. It publishes nothing when any of these fails:

The declaration.

  • declared.json is malformed: a path not normalized, with a .. segment, or outside its place; two outputs on one path; a passage or block declaration on a file that is not rendered; an identity string, a chrome path or a front door missing; a link preview missing a key, with an origin that is not a bare lowercase https origin, with an addresses entry that is not a declared front door or not a directory address, or with an image that is missing, not served, not a PNG, smaller than 1200 × 627, or 5 MiB or larger; or an authored home page that is missing, is not the site's root index.html, or is declared beside a generated one.
  • read/ exists in the source tree.
  • A Markdown file anywhere in the repository, outside hidden directories and _site/, has no declared place: rendered, a redirect, or held with a reason.

Each document.

  • It does not have exactly one top-level H1, coming first.
  • A relative link, anchor or image does not resolve in the source tree, or a link is root-absolute. A path resolves only if every segment matches its directory entry exactly, case included: a clone on a case-insensitive volume would otherwise accept a link the deployment serves as 404.
  • An image names a remote file.
  • Raw HTML is anything but the allowlisted image markup: lowercase <img> tags whose only attributes are a double-quoted src naming a hosted .png file in the site and a double-quoted alt, each at most once, separated only by spaces, tabs or newlines.
    • Every other element, end tag and attribute fails, style and on* event handlers included.
    • So do comments, CDATA sections, declarations, processing instructions, a src with a URL scheme, and raw text containing <!, <? or </script in any case.
    • HTML written inside a code span or a fenced block is literal text, not raw HTML.
  • A blockquote has no declared role, or a passage or block declaration does not name exactly one blockquote or block.
  • A declared structured block does not fit its shape.
  • A table header holds a link, an image or raw HTML. At narrow widths the header row is hidden, so it would hold an interaction the reader cannot see.

The front doors and the carriers.

  • An authored front door does not open from a clone. Outside a build-only fragment, a relative link (<a>, <area>) or resource reference (href, src, srcset) does not resolve, case-exactly, in the source tree, or a link is root-absolute. A root-absolute resource, such as the deployment-root icons, is the site's and is not checked here. A build-only fragment that opens inside another or never closes fails too.
  • The front doors' source links, together, do not name exactly the declared source files: a declared source is not linked, or a .js file is linked that is not declared.
  • The composed front doors do not link each declared source through its source view (read/source/<file>.html), link a page in that directory that maps to no declared source, or link a .js file directly, as a raw link in a build-only fragment would. A root-absolute link is resolved from the site root for this check. The raw file and GitHub are reached from the view.
  • A composed front door leaves a link unclosed, or opens one inside another: a build-only fragment that keeps only one of a link's two tags would do this.
  • An authored front door's own head does not carry the declared link preview: one canonical address, the front door's public address on the declared origin (a directory's index.html by its directory, unless link_preview.addresses names another), with og:url the same; each Open Graph and Twitter key once, with the declared site name, image and alt text, the image's type and dimensions, and the page's own non-empty title and description; no other Open Graph or Twitter key; and none of these tags after the head.
  • A carrier a reading page loads does not match its declared sha256.

After composing. It then composes the site in a staging directory beside --out. It re-reads every written file against its source, and each front door against its composed bytes, and runs the link check over the staging tree:

  • every relative href, src and srcset on a generated page, or a composed front door that stays inside the site must exist, case-exactly. One on a generated page that leaves the site is reported as a warning; one on a composed front door fails, because the deployment would not serve it;
  • every #fragment must name an id in its target page;
  • a GitHub fragment must name a heading of that file;
  • no link may claim a fragment in a raw Markdown file;
  • a composed front door may not link a rendered document as raw .md.

Only then does the staging tree become --out.

The declaration

declared.json lists:

  • identity // the repository's name in the breadcrumb and page titles, the documents index's copy, and the front link the pages' footers and navigation name (the pattern previews);
  • chrome // the stylesheets and scripts every generated page loads, in order, and the wordmark pair: the owner files themselves, at their own paths;
  • documents, the patterns collection and records — what is rendered: the README, CONSUMERS.md, the four semantic primitives, each pattern's contract and hard-fail checklist, the tests README, and this build's own README. A pattern's page sits under read/patterns/, under the output name its entry declares;
  • redirects and held — what is not rendered, each with its target or reason;
  • source_views // empty: no front door links a source file;
  • site_exclude // what the repository root holds that is not site: vercel.json, and _site/, where the build writes;
  • front_doors // the authored pages the build composes: index.html, patterns/index.html and preview/styleguide.html;
  • link_preview // the site's public origin and name, and its social image, a site path, with that image's alt text. Every generated page's head names its canonical address on that origin, a directory's index.html by its directory, and carries Open Graph and Twitter large-image card metadata for the image, with the page's own title and description. The build reads the image's type and dimensions from the PNG itself. An optional addresses object maps a front door to the public address a deployment rewrite serves it at: here preview/styleguide.html at styleguide/. Each authored front door carries the same set in its own head, which the build checks;
  • home_page // the authored index.html, which the build serves as the home page;
  • passages // the role of each blockquote: the README's foundation statement and each status or provenance note are callouts;
  • blocks // the shape of each structured preformatted block.

Passages. A blockquote carries no signal of whose words it holds, so each one is declared by the words it starts with, which must start exactly one blockquote in its document:

role meaning markup
rolequotation meaningrepresented voice: someone's words, quoted or introduced markupblockquote.doc-quote
rolecallout meaningthe document's own thesis, question or contrast, set apart markupp.doc-body.surface-emphasis-rail, or div.doc-group.surface-emphasis-rail around several blocks
roleframing meaningthe statement that opens a whole part before its sections markupthe flat panel with a magenta-bordered chip, its word in chip
rolesynthesis meaningthe closing compression of a part markupthe same panel, its word in chip

Blocks. Every fenced block is a plain .doc-pre, its text as written, unless blocks declares it a grouped diagram. Such a block is declared by the start of its first line, which must start exactly one block in its document, and by its shape:

  • groups-rail // peer groups between blank lines. The unindented lines open each group, and the indented lines beneath them sit one hierarchy rail in, with their shared indentation removed. Any further indentation stays as written.
  • groups-aligned // peer groups between blank lines, each opening on its first line. The lines after it keep every space, because their indentation aligns columns rather than marking a level.

Successive groups sit one line apart. A shape is never inferred from blank lines or capitals: an undeclared block stays plain.

link in a rendered document on the page
link in a rendered documenta rendered document, collection entry or record, with or without #fragment on the pageits page, fragment kept
link in a rendered documenta #fragment into a Markdown file that is not rendered on the pagethe file on GitHub at main, with the anchor
link in a rendered documenta Markdown file in the site that is not rendered on the pagethe file on GitHub at main; its raw route stays served
link in a rendered documentanother file in the site (a figure, a raster, a source file) on the pagea relative link to that file
link in a rendered documenta directory in the site with no index.html on the pagethe directory on GitHub at main
link in a rendered documenta hidden file, or a file the declaration excludes from the site on the pagethe file on GitHub at main
link in a rendered documenta repository file or directory outside the site that is not rendered on the pagethe file or directory on GitHub at main
link in a rendered documentan external URL on the pageunchanged

Emitted paths are percent-encoded. Heading ids follow GitHub's slugs. Headings move down one level so the Markdown H1 sits under the page's breadcrumb title: H1 to H4 render as h2 to h5, and H5 and H6 both render as h6. An image is contained to the column, keeps its aspect ratio and links to its full-resolution file, unless it already sits inside a link. A Markdown file's raw route is its own path on this site.

Carriers

The reading pages load the owner's own files by same-repo reference — colors_and_type.css, surface-panel.css, surface-action.css, surface-text-link.css, surface-document.css, surface-treatments.css, surface-document-overflow.js, patterns/surface-shell/ and the wordmark pair under assets/ — so there is no vendored copy, no pin and no declared hash: carriers declares no vendored or reused file. An owner change reaches the pages at the next deployment.