GuildCommerce Runbook¶

Status: Pre-formation operational draft. The Sysadmin and Full-stack Dev roles will adopt and amend this runbook. Until then, the Administrator runs the platform per this document.

This is the operator-side companion to tech/guildcommerce.md (architecture/showcase) and tech/systems-inventory.md (the canonical inventory). Where they describe what the platform is, this runbook describes how to operate it — deploys, backups, restore drills, on-call, renewals, and the dial-911 sequence.

Scope¶

This runbook covers the GuildCommerce platform stack: the WordPress install at ganjierguild.com, the Moodle install at learn.ganjierguild.com, and the supporting infrastructure (Cloudways Autonomous, Cloudflare, Render workers, GitHub Actions, Workspace Drive). It does not cover policy questions, conduct or sanctions procedures, or anything else routed through counsel.

Component map¶

Component	Hosting	Owner role	Critical?
`ganjierguild.com` (WordPress + BuddyBoss + WooCommerce)	Cloudways Autonomous	Sysadmin	Yes
`learn.ganjierguild.com` (Moodle)	Cloudways Autonomous (co-located)	Sysadmin	Yes
`guildcommerce-theme`	GitHub (private) → deployed via Action	Full-stack Dev	Yes
`guildcommerce-core-plugin`	GitHub (private) → deployed via Action	Full-stack Dev	Yes
Cloudflare (DNS, WAF, cache)	Cloudflare account	Sysadmin	Yes
Stripe + Xsolla	Vendor accounts	Administrator	Yes
Klaviyo	Vendor account	Marketing Lead	No (degraded if down)
Workspace Drive (Restricted recordings + ACLs)	Google Workspace	Administrator	No (degraded if down)
Render workers (queue / Node services)	Render	Full-stack Dev	No
GitHub Actions (PDF build, deploy)	GitHub	Full-stack Dev	No (deferrable)
Document library sync (`gc_doc` CPT)	WP-Cron daily; consumes `dist/library-manifest.json` from this repo	Full-stack Dev	No (degraded if down)

"Critical" = a failure visibly degrades the member-facing experience. The first three rows are the on-call surface.

Environments¶

Environment	URL	Purpose	Refresh from prod?
Production	`ganjierguild.com`, `learn.ganjierguild.com`	The thing members use	n/a
Cloudways sandbox	Cloudways-issued sandbox URL (current home of the in-flight `guildcommerce-theme` + `guildcommerce-core` build)	Where the new theme + plugin stack is being staged for go-live; first surface to verify any new component (Featured Reel, library `doc` post type, etc.)	Manual refresh from prod when needed
Staging	`staging.ganjierguild.com`, `staging.learn.ganjierguild.com`	Pre-deploy verification on the production-mapped staging hostnames (used after the new stack replaces YOOtheme on production)	Weekly automatic; ad-hoc on demand
Local	`*.test` via DDEV or Local	Theme/plugin development	Manual via `phase3-deploy.sh --pull`

Until the new stack go-live cutover, the Cloudways sandbox is the truth surface for the new theme + plugin. staging.ganjierguild.com still mirrors current YOOtheme production and is used for production-side hotfix testing only. After cutover, the Cloudways sandbox returns to its general-purpose role and staging.* becomes the canonical pre-deploy environment for the new stack.

The shared asset staging-guard.php is a must-load in both staging environments — it adds the [staging] banner, blocks transactional emails, and disables Stripe/Xsolla webhook outbound calls. Production must never load staging-guard.php. Confirmed by the deploy scripts.

Deploys¶

Theme (`guildcommerce-theme`)¶

Open PR against main of guildcommerce-theme repo.
CI runs lint, build, visual-regression on the showcase pages.
Merge to main triggers theme-deploy.yml:
Builds guildcommerce-theme.tar.gz
Pushes to staging via Cloudways API
Smoke check (4 URLs render 200; logo present)
Operator acks staging in #deploys channel
Manual promote step → production
After production deploy: smoke check + cache purge on Cloudflare for the theme asset paths.
Tag release in GitHub.

Rollback: Re-run the workflow against the previous green tag. The shared asset gc-theme-failover.php is the emergency white-label fallback if the active theme directory is corrupted; activating it requires SSH access to the Cloudways environment and is documented in the dial-911 sequence below.

Core plugin (`guildcommerce-core-plugin`)¶

Same shape as the theme: PR → CI → merge → staged deploy → smoke check → manual promote. The deploy artifact is guildcommerce-core.tar.gz. Activation/deactivation in WP-Admin is handled by the deploy script, not by hand.

Migrations: Schema changes live in includes/migrations/ and run on plugin activation. Every migration is idempotent (run-safe twice). Before deploying a migration to production: confirm staging carries a recent prod-data refresh and the migration completes there cleanly.

Document library sync (`gc_doc` CPT) — ADR-0010¶

The Library module in guildcommerce-core mirrors this repo's Markdown library into WordPress as a gc_doc custom post type. Members read library docs through the BuddyBoss portal section on guildcommerce.pplx.app; operators/board read them through the Hub area on the new WP install. Both surfaces consume the same library-manifest.json; per-doc visibility comes from each doc's frontmatter visible_to list (Form O slugs).

Source of truth: dist/library-manifest.json in this repo, built nightly by .github/workflows/build-pdfs.yml (the build-library-manifest.py step) and uploaded as the ganjier-guild-library-manifest artifact. The plugin reads the raw GitHub URL on main by default.

Sync mechanics:

Daily WP-Cron event gc_library_sync (registered on plugin load).
Sync upserts by doc_id with a manifest_hash change-detection short-circuit (unchanged docs do not re-write).
Docs that disappear from the manifest are drafted, never deleted (preserves edit history and avoids accidental data loss).
Per-doc visibility is computed from frontmatter visible_to and persisted as gc_acl_tier meta. The existing 4-tier ACL (PUBLIC / MEMBER / BOARD / PRIVILEGED) gates the_content, REST, search, and Woo Memberships — gc_doc is in the gated allow-list.
Form O slug → tier map (more permissive wins): guest → PUBLIC; member/ganjier/navigator/voyage-crew/voyage-captain/coop-participant/contractor/employee/sponsor → MEMBER; operator/board/officer/committee-chair → BOARD; sysadmin/counsel only OR audience: restricted → PRIVILEGED.

Config knob: override the manifest URL with the GUILDCOMMERCE_LIBRARY_MANIFEST_URL constant in wp-config.php (e.g. point a sandbox at a feature-branch raw URL) or the guildcommerce_library_manifest_url filter for runtime overrides.

Operator commands (run on the Cloudways sandbox or wherever the plugin is installed):

# Manual rerun — useful right after merging a docs PR or when the daily cron is sleeping
wp guildcommerce library sync

# Point at a feature branch for a single run (does not persist)
wp guildcommerce library sync --url=https://raw.githubusercontent.com/ginalyncox/ganjier-guild-launch-blueprint/<branch>/dist/library-manifest.json

# Show last sync time, manifest URL in use, and counts of published vs draft gc_doc posts
wp guildcommerce library status

There is also a REST endpoint POST /gc/v1/library/sync (gated to manage_options) for triggering a sync from automation.

Verification on the Cloudways sandbox after deploy:

wp guildcommerce library status — confirms manifest URL points at this repo's raw main URL and that a recent sync timestamp is set.
wp guildcommerce library sync — should report a non-zero fetched count and zero errors.
Visit /wp-admin/edit.php?post_type=gc_doc — confirms ~70+ docs are present (current manifest is 72 docs).
Spot-check a member-tier doc as a logged-out user — should be gated; as a member — should render.

Failure modes & first response:

Sync errors out → check wp guildcommerce library status summary; if the manifest URL is unreachable, fall back to a feature-branch URL via --url. The cron will retry the next day automatically.
Manifest hash drift but no content change → harmless; on the next run it reconciles.
Doc removed from manifest but still visible to members → run wp guildcommerce library sync; the orphan should drop to draft on the next reconciliation.

WordPress core / BuddyBoss / WooCommerce / Stripe Connect¶

Major-version upgrades follow the same path: staging first, smoke check, scheduled production window. Patch-level upgrades for security may go straight to production after a quick staging spot-check.

WP-cron: the cron should be system-driven, not request-driven. Confirmed by:

define('DISABLE_WP_CRON', true);

in wp-config.php plus a Cloudways scheduled task hitting wp-cron.php every 5 minutes.

Moodle¶

Moodle upgrades follow Moodle's standard upgrade flow on Cloudways. Specific to the Guild's setup:

Backup database + moodledata (full, not just managed-backup snapshot).
Put Moodle into maintenance mode.
Run upgrade.
Test the SSO bridge (OIDC) end-to-end with a test member account.
Test the CE webhook by completing a test course → confirming ce-ledger row written in guildcommerce-core-plugin.
Take Moodle out of maintenance mode.

mod_customcert and the Boost Union theme are pinned to Guild-tested versions in tech/systems-inventory.md. Pin updates ride the major-Moodle-upgrade train.

Phase scripts¶

The shared deploy scripts (phase3-deploy.sh, phase3b-deploy.sh) live at the repo root in the guildcommerce repo. They wrap the Cloudways API + GitHub releases. Read the script header before each run; flags can change between phases.

Backups¶

Tier	What	Cadence	Retention	Where
App-level	Cloudways managed backups (full app + DB)	Daily	14 days	Cloudways block storage
DB extracts	`wp_*` tables + Moodle DB nightly dumps	Daily	30 days	Cloudways block storage
Off-site	Encrypted weekly snapshot of both apps + DBs	Weekly	90 days	S3 bucket (separate account)
Long-term	Quarterly archive snapshot	Quarterly	7 years	S3 Glacier
Forms PDFs	Built artifacts from `.github/workflows/build-pdfs.yml`	Per push	90 days runs / indefinite tags	GitHub Actions artifacts + GitHub Releases
Drive (Restricted recordings)	Workspace native versioning + Vault retention rules	Continuous	Per the resource's `retention_years` (typically 7+)	Google Workspace

Encryption keys for the off-site snapshots are held by the Sysadmin in the Guild's password manager with a break-glass copy in the Administrator's Form K — Emergency Access & Recovery Record.

Restore drills¶

Quarterly the Sysadmin restores from each tier into the staging environment and verifies a member account can log in, view a course, and download a form. The drill is logged in Form K. A drill that takes longer than the recovery objective below triggers a re-architecture conversation with the Full-stack Dev.

Recovery target	RTO (time to recover)	RPO (max data loss)
WordPress + BuddyBoss member experience	4 hours	24 hours
Moodle courses + gradebook	8 hours	24 hours
Forms library	1 hour (rebuild from repo)	0 (source in git)
WooCommerce orders + Stripe ledger	1 hour	0 (Stripe is the source of truth; WP is a mirror)

On-call¶

The Guild does not run a 24/7 on-call. Member-visible production issues during daytime business hours (9a–6p Central) are responded to within 2 hours. Outside those hours: best effort, status-page update only.

Tier	Owner	Backup	Escalation
First responder	Administrator	Sysadmin	If platform-wide outage, escalate to Sysadmin within 30 min
Platform incident	Sysadmin	Full-stack Dev	If Cloudways or DB is down, page Cloudways support; if more than 60 min, post status page update
Code/data incident	Full-stack Dev	Sysadmin	If suspected data loss or corruption, freeze writes (maintenance mode) before continuing

Status page lives at status.ganjierguild.com (Cloudflare-hosted static page). Updates are posted by the first responder or Sysadmin; never by a sponsor, partner, or member.

Renewals & cost-events¶

Tracked in the Administrator's calendar. Each line item has the renewal date, who owns the relationship, and what the auto-renew default is.

Item	Renewal cadence	Owner	Auto-renew?
Domain `ganjierguild.com` (and other Guild-owned domains)	Annual	Administrator	Yes; multi-year recommended
Cloudways Autonomous	Monthly	Administrator	Yes
Cloudflare (Pro or higher tier if used)	Annual	Sysadmin	Yes
Stripe (no fixed fee, transaction-based)	n/a	Administrator	n/a
Xsolla (no fixed fee, transaction-based)	n/a	Administrator	n/a
Klaviyo subscription	Monthly or annual depending on plan	Marketing Lead	Yes
Google Workspace seats	Per-seat monthly or annual	Administrator	Yes
BuddyBoss Platform plugin	Free (open source)	n/a	n/a
WooCommerce Subscriptions	Annual	Full-stack Dev	Yes
Moodle plugins (`mod_customcert`, Boost Union, OIDC)	Free (open source)	n/a	n/a
Adobe Fonts kit `cuv7zfu`	Bundled with GoDaddy account	Administrator	Yes (with GoDaddy renewal)
S3 off-site backup bucket	Pay-per-use	Sysadmin	n/a

Renewals at risk (CC expiring, plan change required, vendor pricing change) get logged in the next finance committee report.

Quarterly access review¶

Once per quarter the Sysadmin runs the access review across the platform:

WP-Admin user list vs current staff/volunteer roster — anyone who's offboarded gets removed (referenced via Form J).
Cloudways environment SSH/SFTP access list vs current Sysadmin + Full-stack Dev roster.
Moodle site administrator list — should be the Sysadmin only, plus a board-level break-glass.
Cloudflare account members vs current roster.
Workspace Drive folder ACLs for Restricted folders vs Form O.
Stripe + Xsolla team-member access vs current roster.
GitHub access to private repos (guildcommerce, guildcommerce-theme, guildcommerce-core-plugin) vs current roster.

Findings logged in Form K. Anything that requires removing access from a person who is currently active in another capacity is escalated to the Administrator + Operations Lead before action.

Dial-911 sequence¶

When the member-visible site is down or actively serving wrong data. The order matters — do not skip steps.

Confirm the outage from a clean network (mobile data, not office Wi-Fi). Open ganjierguild.com and learn.ganjierguild.com in a private window, no extensions.
Check Cloudflare — is the origin reachable? If Cloudflare is masking a 5xx from origin, there's a banner in the Cloudflare dashboard.
Check Cloudways — environment status page. If Cloudways is in degraded state across multiple customers, post status-page update and wait for Cloudways.
Check the deploy log — was a theme or plugin deploy in flight? If yes, roll back to the previous green tag using the deploy workflow.
Maintenance mode — if the issue is data integrity (members reporting wrong dues status, wrong CE counts, missing course access), put the affected app into maintenance mode before continuing. WP: wp maintenance-mode activate. Moodle: site admin → maintenance mode.
Theme failover — if the symptom is "site renders broken or unstyled," activate gc-theme-failover.php from the SSH session per its header doc. This swaps to a minimal white-label theme so members can still log in and complete urgent transactions while the team fixes the active theme.
Restore from backup — only after maintenance mode is on. Restore order: DB first, then files. Verify with a test member login before exiting maintenance mode.
Post-mortem — within 7 days of recovery, the Sysadmin writes a one-page post-mortem (what happened, contributing factors, what changes — no individual blame). File in the Sysadmin's incident folder; summarize in the next board report.

What this runbook does NOT cover¶

These are deliberately routed elsewhere:

Conduct or sanctions procedure — counsel-drafted policy via the Ethics Committee
Member dispute resolution about credentialing — appeals & revocation governs that path
Privacy-incident notification (data breach response) — counsel-drafted; the runbook may detect the technical event but the notification flow is owned by counsel and the board
Financial reconciliation — finance/ governs the books; this runbook may produce evidence (logs, exports) but does not interpret it
Brand-license interpretation — ADR-0008 and counsel