Related Pages

Related topics: Lib

Overview

PortalJS is an open-source framework for building modern data portals. Originally forked and rebranded from the Recline.js data explorer, it has grown into a full-stack toolkit that covers portal scaffolding, frontend data exploration, CKAN-compatible APIs, authentication, background processing, and Git LFS object storage. The repository is organized as a monorepo that groups several independently versioned packages under one source tree.

Purpose and Scope

The repository serves two distinct but related audiences:

• Data portal builders who want to launch an end-user data portal with dataset catalogs, previews, charts, and search.
• Platform operators who need the backend services that power those portals: a REST API, an auth layer, background workers, and large-file storage.

According to README.md, PortalJS positions itself as a "framework for creating data portals" with components, examples, and integrations spanning the full stack. The community has confirmed the migration track from older CKAN / Datahub templates to Next.js (App Router) for portal UIs, as tracked in issues/1618.

Repository Layout

The top-level directories each correspond to a deployable unit:

• cloud/api/ — the public REST surface used by portals. Source: cloud/api/package.json:1-40 names this package @portaljs/cloud-api and wires Express, CKAN-compatible endpoints, and Prisma for persistence.
• cloud/auth/ — authentication service exposed as @portaljs/cloud-auth. Source: cloud/auth/package.json:1-30 shows NextAuth integration with Prisma adapters and OAuth providers.
• cloud/worker/ — a worker process for async jobs (indexing, exports). Source: cloud/worker/package.json:1-30 declares a BullMQ-based queue consuming from Redis.
• giftless/ — a standalone Git LFS server used to back large dataset files. Source: giftless/README.md:1-60 describes its role as the storage layer behind the portal's data APIs.
• giftless/cloudflare/ — a Cloudflare Workers deployment variant of Giftless. Source: giftless/cloudflare/README.md:1-40 documents the edge runtime, Durable Object backends, and KV/WebAssembly storage adapters.

A simplified view of how these pieces relate:

flowchart LR
    Browser[Portal Frontend] --> API[cloud/api]
    Browser --> Auth[cloud/auth]
    API --> DB[(Postgres via Prisma)]
    API --> Giftless[giftless LFS]
    Worker[cloud/worker] --> DB
    Worker --> Giftless
    Giftless --> Storage[(Object Storage)]

Component Highlights

Portal Frontend

The user-facing portal is built on Next.js and reuses @portaljs/components for dataset tables, charts, and map previews. Recent component releases focus on robustness — for example, @portaljs/[email protected] "breaks lines at invalid / missing values" in LineChart, and @portaljs/[email protected] adds yearmonth to xAxisTimeUnit. The community also tracks PDF styling fixes (e.g., @portaljs/[email protected]).

CKAN API Client

@portaljs/[email protected] provides a typed client for CKAN endpoints. This is the contract that lets PortalJS Cloud interoperate with existing CKAN integrations without breaking downstream consumers, a point emphasized in the FAQ added under issues/1502.

Auth, Worker, and Storage

• cloud/auth handles session management and provider callbacks.
• cloud/worker offloads long-running operations such as dataset indexing and exports to a queue.
• giftless implements the Git LFS protocol so portals can version and stream large files; the Cloudflare variant runs the same protocol on the edge with Durable Object state.

Scaffolding and Skills

create-portaljs is the official scaffolder (npm create portaljs@latest). Recent releases bundle agentic "skills" that guide AI assistants when bootstrapping or extending a portal. [email protected] standardized the portaljs- prefix for all skills (e.g., /portaljs-new-portal, /portaljs-add-dataset); earlier releases introduced /migrate (@0.5.0), bundled skills into .claude/ (@0.4.0), and added the branded cyclone loader (@0.3.0).

Community Signals

Community activity aligns with the architecture above:

• issues/520 traces the historical rename of Recline.js to Portal.JS, framing it as a modern React-based data portal framework.
• issues/1138 discusses how LLMs could automate schema work — a direction now reflected in the agentic skills shipped with create-portaljs.
• issues/1059 surfaces a remark-wiki-link bug, indicating the docs site reuses Markdown pipelines that PortalJS contributors should be aware of.
• issues/1095 and issues/819 are operational issues (a 404 on an airports code list, and a stale xlsx / SheetJS registry) that illustrate the kind of dataset-quality friction portals must handle.

Together, the packages, scripts, and community tracks make PortalJS a full-stack framework rather than a single library.

Related Pages

Related topics: Overview, Lib

Lib

The site/lib/ directory in the PortalJS repository hosts the server-side and client-side helper modules used by the marketing and documentation site (Next.js app) that lives under site/. It is not the core portal framework itself (which is provided by @portaljs/components and @portaljs/ckan-api-client-js); rather, it is a thin utility layer that exposes configuration constants, content-fetching helpers, analytics wiring, and small transformation functions reused across pages, components, and static generation routines.

Scope and High-Level Role

The files in site/lib/ fall into four functional groups:

1. Shared constants and primitives — common.ts exports site-wide configuration values used by both client and server contexts.
2. Data transformation utilities — computeFields.ts provides pure functions that reshape raw records into computed fields before they are rendered.
3. Remote data fetchers — getGitHubData.ts, getAuthorsDetails.ts, and getPageData.js fetch external state (GitHub metadata, author bios, CMS-driven page content) at build time or request time.
4. Analytics integration — gtag.js integrates Google Analytics, initializing the tracker and exposing helpers for page views and events.

Together, these modules decouple page-level code from configuration, HTTP concerns, and analytics plumbing so that UI components can focus on rendering.

Source: site/lib/common.ts and site/lib/getPageData.js act as the structural backbone for the rest of the site.

Configuration and Shared Constants

site/lib/common.ts centralizes constants such as the GitHub repository coordinates used by social cards and "star us" CTAs, the canonical site URL, Open Graph defaults, and any feature flags consumed by multiple pages. Centralizing these values prevents drift between server-rendered HTML, metadata, and the client-side runtime; when the project URL or repo slug changes, only this file needs editing. Both getGitHubData.ts and getAuthorsDetails.ts read from common.ts so that the GitHub owner/name pair stays consistent across contributors listings and star-count widgets.

Source: site/lib/common.ts defines the canonical constants consumed by every fetcher and UI component in the site.

Data Fetching Helpers

The fetcher layer is split by responsibility to keep error handling local and to make each helper independently mockable in tests.

getGitHubData.ts wraps the GitHub REST API (repository metadata such as stargazers_count, default branch, and license) and returns a normalized object that downstream components consume without re-parsing the raw response. It is invoked during static generation for the homepage and the contributors page so that the marketing surface always shows up-to-date numbers without making client requests.

getAuthorsDetails.ts complements the GitHub fetcher: for each contributor handle it resolves a richer profile (avatar URL, bio, social links) so the page can render author cards alongside blog posts and changelog entries. Centralizing the resolution makes it easy to swap data sources (e.g. switching from GitHub to a CMS) without touching every page.

getPageData.js is the generic CMS page resolver used by the documentation, blog, and changelog routes. It accepts a slug and a content type, calls the configured headless CMS endpoint, and returns the parsed payload. Because it is the single entry point for content retrieval, error logging, caching headers, and TypeScript/JSDoc type guards are written once and reused everywhere, which is consistent with PortalJS's modular packaging philosophy visible in releases such as @portaljs/[email protected].

Source: site/lib/getGitHubData.ts, site/lib/getAuthorsDetails.ts, and site/lib/getPageData.js form the three fetchers used by static and dynamic routes.

Transformations and Analytics

computeFields.ts is a small, side-effect-free transformer used after a fetcher returns raw data. For example, given a blog post payload it can compute reading time, formatted publish dates, and trimmed excerpts from a longer body. Keeping this logic out of page components makes it straightforward to unit test the transformation in isolation and to reuse it across different routes (the blog index, post detail page, and RSS feed all share the same computeFields pipeline).

gtag.js is the analytics layer. It initializes gtag('js', new Date(), 'G-XXXX'), exposes a pageview helper that the Next.js _app or app/ router invokes on route change, and provides a typed event helper for tracking CTA clicks on the homepage and documentation CTAs. The file is deliberately framework-agnostic so it can be reused regardless of whether the site is rendered with the Pages Router or the App Router that the research.portaljs.com migration referenced in community issue #1618 adopts.

Source: site/lib/computeFields.ts keeps rendering logic separate from data logic, while site/lib/gtag.js wires up analytics.

Architectural Sketch

flowchart LR
  A[Page / Route] --> B[common.ts constants]
  A --> C[getPageData.js]
  A --> D[getGitHubData.ts]
  A --> E[getAuthorsDetails.ts]
  C --> F[computeFields.ts]
  D --> F
  E --> F
  F --> G[UI Components]
  A --> H[gtag.js]
  H --> I[Google Analytics]

Pages pull constants from common.ts, fetch content through one of the typed fetchers, normalize the result via computeFields.ts, render it with shared UI components, and report engagement through gtag.js.

How to Extend

When adding a new helper, prefer placement in site/lib/ rather than inside a component file so it can be imported by both server and client code paths without bundling accidental dependencies. Follow the existing convention of one concern per file (fetchers separated from transformers, analytics isolated from data), match the typed signatures used by getGitHubData.ts and computeFields.ts, and register any new constant in common.ts to keep configuration discoverable.

Source: directory conventions observed across site/lib/common.ts, site/lib/computeFields.ts, site/lib/getGitHubData.ts, site/lib/getPageData.js, and site/lib/gtag.js.

Related Pages

Related topics: Lib, Src

Src

The src directory is the canonical implementation root for every package published from the datopian/portaljs monorepo. While portaljs exposes several npm-facing packages (@portaljs/core, @portaljs/components, @portaljs/ckan-api-client-js, create-portaljs), each of them shares the same convention: a top-level src/ folder holds the TypeScript/JavaScript sources that are then compiled into the published dist/ artifacts. This page describes the cross-package shape of those src/ trees, the conventions they follow, and how they map to the surfaces consumed by users.

Monorepo Layout and Package Boundaries

The monorepo is organised under packages/, and each workspace owns an independent src/ tree. The three library packages plus the CLI scaffolder are the units a contributor will most often interact with:

• packages/core/src/ — runtime configuration, public type declarations, and the embedded UI primitives that ship with @portaljs/core. Source: packages/core/src/index.ts:1-40
• packages/components/src/ — React building blocks (charts, maps, viewers, dataset components) re-exported as @portaljs/components. Source: packages/components/src/index.ts:1-30
• packages/ckan-api-client-js/src/ — typed CKAN API client that backs PortalJS Cloud integrations. Source: packages/ckan-api-client-js/src/index.ts:1-50
• create-portaljs/src/ — implementation of the npm create portaljs@latest scaffolder used to bootstrap new data portals. Source: create-portaljs/src/index.ts:1-60

The src/ folders are deliberately kept thin: each one re-exports a curated public API, with the heavy lifting delegated to subdirectories such as ui/, config/, and types/ within the same package root.

`packages/core/src/` — Configuration, Types, and Embedded UI

The core package is the most layered of the four. Its src/ tree has three responsibilities:

1. Public entrypoint. index.ts aggregates the runtime exports (configuration helpers, theme tokens, and the small UI set) and is what consumers import when they write import { ... } from '@portaljs/core'. Source: packages/core/src/index.ts:1-40
2. Configuration defaults. The config/ subdirectory ships opinionated defaults and a re-export index. default.ts defines the baseline settings (theme palette, layout primitives, plugin slots) that downstream portals extend, and index.ts exposes the typed accessors. Source: packages/core/src/config/default.ts:1-120 and Source: packages/core/src/config/index.ts:1-20
3. Embedded UI primitives. ui/ hosts lightweight, framework-agnostic React components used by every portal built on top of @portaljs/core. Each primitive lives in its own folder with a colocated index.ts barrel. For example, Avatar/Avatar.tsx defines the component, and Avatar/index.ts re-exports it. Source: packages/core/src/ui/Avatar/Avatar.tsx:1-60 and Source: packages/core/src/ui/Avatar/index.ts:1-10

Type definitions shared across the runtime live in packages/core/src/types/index.d.ts, an ambient declaration file that augments the public surface with the interfaces downstream portals rely on (theme shape, plugin contracts, dataset rows). Source: packages/core/src/types/index.d.ts:1-80

`packages/components/src/` — Composable Data Surface

The components package mirrors core's barrel convention but is focused on the data-experience layer: tables, viewers, charts, and the dataset detail shell. The single index.ts barrel re-exports each component module so that consumers can import them by name. Releases such as @portaljs/[email protected] (LineChart gap handling), 1.2.2 (xAxisTimeUnit: 'yearmonth'), and 1.2.1 (optional tileLayerName for Map) are patched in this src/ tree. Source: packages/components/src/index.ts:1-30

`packages/ckan-api-client-js/src/` — Typed CKAN Client

The @portaljs/ckan-api-client-js package (initial public release 1.4.0) is the typed bridge to CKAN's HTTP API. Its src/ is split between the client implementation in index.ts and the request/response model declarations in types.ts. The client is what makes the FAQ claim — *"PortalJS Cloud is built on CKAN's APIs, so all standard CKAN endpoints continue to work after migration"* — observable in code: consumers call methods like packageSearch, groupList, or tagShow and receive strongly-typed results. Source: packages/ckan-api-client-js/src/index.ts:1-50 and Source: packages/ckan-api-client-js/src/types.ts:1-100

`create-portaljs/src/` — Scaffolding Engine

The create-portaljs package implements the CLI that scaffolds a new PortalJS project via npm create portaljs@latest. Its src/index.ts orchestrates template selection, file emission, and post-install hints (e.g., directing users to the bundled .claude/ agentic skills introduced in 0.4.0). Recent releases — 0.6.0 (renaming the skill suite to the portaljs- prefix), 0.5.0 (bundling the /migrate skill), 0.4.0 (bundling agentic skills into .claude/), 0.3.0 (branded cyclone spinner), and the initial 0.2.0 — are all implemented inside this folder. Source: create-portaljs/src/index.ts:1-60

Cross-Cutting Conventions

A few conventions are uniform across every src/ tree in the monorepo:

These conventions are what allow the create-portaljs scaffolder to copy src/-shaped templates verbatim and what keep every release aligned with the public API documented in each package's README. When a community issue references a runtime behaviour — for example, the remark-wiki-link failure in #1059 or the xlsx/SheetJS version note in #819 — the fix invariably lands inside one of these src/ trees and propagates through the corresponding dist/ artifact at publish time.

Doramagic Pitfall Log

Found 9 structured pitfall item(s), including 0 high/blocking item(s). Top priority: Configuration risk - Configuration risk requires verification.

1. Configuration risk: Configuration risk requires verification

• Severity: medium
• Finding: Project evidence flags a configuration risk. Review the linked source before relying on this workflow.
• User impact: May increase setup, validation, or first-run risk for the user.
• Recommended check: Reproduce the official install and quickstart path in an isolated environment.
• Evidence: capability.host_targets | https://github.com/datopian/portaljs

2. Capability evidence risk: Capability evidence risk requires verification

• Severity: medium
• Finding: README/documentation is current enough for a first validation pass.
• User impact: May increase setup, validation, or first-run risk for the user.
• Recommended check: Reproduce the official install and quickstart path in an isolated environment.
• Evidence: capability.assumptions | https://github.com/datopian/portaljs

3. Runtime risk: Runtime risk requires verification

• Severity: medium
• Finding: Project evidence flags a runtime risk. Review the linked source before relying on this workflow.
• User impact: May increase setup, validation, or first-run risk for the user.
• Recommended check: Reproduce the official install and quickstart path in an isolated environment.
• Evidence: community_evidence:github | https://github.com/datopian/portaljs/issues/1618

4. Maintenance risk: Maintenance risk requires verification

• Severity: medium
• Finding: Project evidence flags a maintenance risk. Review the linked source before relying on this workflow.
• User impact: May increase setup, validation, or first-run risk for the user.
• Recommended check: Reproduce the official install and quickstart path in an isolated environment.
• Evidence: evidence.maintainer_signals | https://github.com/datopian/portaljs

5. Security or permission risk: Security or permission risk requires verification

• Severity: medium
• Finding: no_demo
• User impact: May increase setup, validation, or first-run risk for the user.
• Recommended check: Reproduce the official install and quickstart path in an isolated environment.
• Evidence: downstream_validation.risk_items | https://github.com/datopian/portaljs

6. Security or permission risk: Security or permission risk requires verification

• Severity: medium
• Finding: no_demo
• User impact: May increase setup, validation, or first-run risk for the user.
• Recommended check: Reproduce the official install and quickstart path in an isolated environment.
• Evidence: risks.scoring_risks | https://github.com/datopian/portaljs

7. Security or permission risk: Security or permission risk requires verification

• Severity: medium
• Finding: Project evidence flags a security or permission risk. Review the linked source before relying on this workflow.
• User impact: May increase setup, validation, or first-run risk for the user.
• Recommended check: Reproduce the official install and quickstart path in an isolated environment.
• Evidence: community_evidence:github | https://github.com/datopian/portaljs/issues/1502

8. Maintenance risk: Maintenance risk requires verification

• Severity: low
• Finding: issue_or_pr_quality=unknown。
• User impact: May increase setup, validation, or first-run risk for the user.
• Recommended check: Reproduce the official install and quickstart path in an isolated environment.
• Evidence: evidence.maintainer_signals | https://github.com/datopian/portaljs

9. Maintenance risk: Maintenance risk requires verification

• Severity: low
• Finding: release_recency=unknown。
• User impact: May increase setup, validation, or first-run risk for the user.
• Recommended check: Reproduce the official install and quickstart path in an isolated environment.
• Evidence: evidence.maintainer_signals | https://github.com/datopian/portaljs

Community Discussion Evidence

Doramagic exposes project-level community discussion separately from official documentation. Review these links before using portaljs with real data or production workflows.

• Feedback on research.portaljs.com - github / github_issue
• Add 6 new FAQs in FAQ page - github / github_issue
• [email protected] - github / github_release
• [email protected] - github / github_release
• [email protected] - github / github_release
• [email protected] - github / github_release
• [email protected] - github / github_release
• @portaljs/[email protected] - github / github_release
• @portaljs/[email protected] - github / github_release
• @portaljs/[email protected] - github / github_release
• @portaljs/[email protected] - github / github_release
• @portaljs/[email protected] - github / github_release