Implementing a Shared Content Repository with Git Submodules and GitHub Actions

Description

This article documents the implementation of a shared content repository architecture using Git submodules and GitHub Actions to manage markdown content across multiple static site deployments. The solution enables centralized content management with automated deployment to multiple frontends.

For the current operator/contributor guide and rendered architecture diagrams, see Shared Content Project Guide.

Architecture Overview

The implementation uses a three-repository structure:

1. Content Repository (wikip-co/content): Central repository containing all markdown files
2. Site Repository 1 (wikip-co/wikip.co): Hexo static site using content as a submodule
3. Site Repository 2 (anthonyrussano/anthonyrussano.com): Hexo static site using content as a submodule

When markdown files are modified in the content repository, GitHub Actions automatically triggers rebuilds of both site repositories.

Implementation Steps

Step 1: Analyze Existing Content

First, compare the content between both repositories to identify differences:

1
2
3
4
5
6
7
8
9

# Count markdown files in each repo
find /home/anthony/wikip.co/site/source/_posts -name "*.md" | wc -l
# Result: 3,188 files

find /home/anthony/anthonyrussano.com/site/source/_posts -name "*.md" | wc -l
# Result: 3,195 files

# Compare directories to find differences
diff -r wikip.co/site/source/_posts/ anthonyrussano.com/site/source/_posts/ --brief

Reasoning: Understanding the differences between repositories ensures no content is lost during the merge. In this case, there were only 8 differences:

• 2 files with different content
• 6 unique files/folders across both repos

Step 2: Create Unified Content Repository

Merge content from both repositories into a single unified collection:

1
2
3
4
5
6
7
8
9
10
11
12
13

# Create unified directory with content from one repo
mkdir -p /home/anthony/content-unified
cp -r /home/anthony/anthonyrussano.com/site/source/_posts/* /home/anthony/content-unified/

# Copy unique files from the other repo
cp -r "/home/anthony/wikip.co/site/source/_posts/Child Development/Parenting" \
     "/home/anthony/content-unified/Child Development/"
cp -r "/home/anthony/wikip.co/site/source/_posts/Current Events/Technology" \
     "/home/anthony/content-unified/Current Events/"

# Resolve conflicts by choosing the better version
cp "/home/anthony/wikip.co/site/source/_posts/Natural Healing/Oils and Fats/coconut-oil.md" \
   "/home/anthony/content-unified/Natural Healing/Oils and Fats/coconut-oil.md"

Reasoning: Starting with the repo containing more files (anthonyrussano.com with 3,195 files) as the base, then adding unique content from the other repo ensures all content is preserved. For conflicting files, manual review determined which version had more complete information.

Final unified count: 3,198 markdown files

Step 3: Create GitHub Repository for Content

Initialize the unified content as a Git repository and push to GitHub:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

cd /home/anthony/content-unified
git init
git config user.email "[email protected]"
git config user.name "Anthony Russano"
git add .
git commit -m "Initial commit: unified markdown content from wikip.co and anthonyrussano.com"

# Create repository using GitHub CLI
gh repo create wikip-co/content --public \
  --description "Shared markdown content for wikip.co and anthonyrussano.com" \
  --source . --remote origin

# Push to GitHub
git branch -M main
git push -u origin main

Reasoning: Using gh CLI streamlines repository creation and automatically configures the remote. The organization account (wikip-co) provides a neutral namespace for shared content accessible to both sites.

Step 4: Configure Submodules in Site Repositories

Replace the _posts directory in each site repository with the content submodule:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

# For wikip.co
cd /home/anthony/wikip.co
git rm -r site/source/_posts
git submodule add https://github.com/wikip-co/content.git site/source/_posts
git add .
git commit -m "Replace _posts with content submodule"
git push

# For anthonyrussano.com
cd /home/anthony/anthonyrussano.com
git rm -r site/source/_posts
git submodule add https://github.com/wikip-co/content.git site/source/_posts
git add .
git commit -m "Replace _posts with content submodule"
git push

Reasoning: Git submodules allow referencing an external repository at a specific commit. This enables both site repos to share the same content while maintaining their own site configurations, themes, and build processes.

Step 5: Configure Submodule to Track Main Branch

Update .gitmodules to track the main branch:

1
2
3
4
5

# Edit .gitmodules in both repos to add:
[submodule "site/source/_posts"]
    path = site/source/_posts
    url = https://github.com/wikip-co/content.git
    branch = main

Reasoning: The branch setting is still useful metadata for local workflows and explicit fetches, but the current deployment flow no longer relies on git submodule update --remote alone. The live build now receives the exact content SHA through repository_dispatch and checks out that specific revision for reproducibility.

Step 6: Create Trigger Workflow in Content Repository

Create a GitHub Actions workflow that triggers both site repositories when content changes:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37

# .github/workflows/trigger-sites.yml
name: Trigger Site Rebuilds

on:
  push:
    branches:
      - main
    paths:
      - '**/*.md'
  workflow_dispatch:

permissions:
  contents: read

jobs:
  trigger-rebuilds:
    runs-on: ubuntu-latest
    steps:
      - name: Trigger wikip.co rebuild
        run: |
          curl -L \
            -X POST \
            -H "Accept: application/vnd.github+json" \
            -H "Authorization: Bearer ${{ secrets.TRIGGER_TOKEN }}" \
            -H "X-GitHub-Api-Version: 2022-11-28" \
            https://api.github.com/repos/wikip-co/wikip.co/dispatches \
            -d '{"event_type":"content-updated","client_payload":{"content_repository":"'"${{ github.repository }}"'","content_ref":"'"${{ github.ref_name }}"'","content_sha":"'"${{ github.sha }}"'","source_run_url":"'"${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}"'"}}'

      - name: Trigger anthonyrussano.com rebuild
        run: |
          curl -L \
            -X POST \
            -H "Accept: application/vnd.github+json" \
            -H "Authorization: Bearer ${{ secrets.TRIGGER_TOKEN }}" \
            -H "X-GitHub-Api-Version: 2022-11-28" \
            https://api.github.com/repos/anthonyrussano/anthonyrussano.com/dispatches \
            -d '{"event_type":"content-updated","client_payload":{"content_repository":"'"${{ github.repository }}"'","content_ref":"'"${{ github.ref_name }}"'","content_sha":"'"${{ github.sha }}"'","source_run_url":"'"${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}"'"}}'

Reasoning: Using repository_dispatch events allows triggering workflows in other repositories. The workflow only runs when .md files change, avoiding unnecessary builds for infrastructure changes.

Step 7: Set Up GitHub Token

Create and configure a Personal Access Token for triggering workflows:

1
2
3
4
5
6

# The existing gh CLI token already had repo scope
# Store it as a secret in the content repository
gh auth token | gh secret set TRIGGER_TOKEN -R wikip-co/content

# Verify the secret was created
gh secret list -R wikip-co/content

Reasoning: The repo scope includes permission to trigger repository_dispatch events. Using the existing gh CLI token avoids creating additional tokens, and storing it as a GitHub Secret keeps it secure.

Step 8: Update Site Repository Workflows

Modify the deployment workflows in both site repositories to:

1. Listen for repository_dispatch events
2. Pass the dispatched content ref/SHA into a reusable deploy workflow
3. Resolve the content submodule to that exact revision before the build starts

For wikip.co (/.github/workflows/generator.yml):

1
2
3
4
5
6
7
8
9

on:
  workflow_dispatch:
  repository_dispatch:
    types: [content-updated]
  push:
    branches:
      - main
    paths:
      - 'site/**'

Update the site workflow to call the reusable deploy workflow:

1
2
3
4
5
6
7
8
9
10

jobs:
  build:
    uses: wikip-co/content/.github/workflows/hexo-deploy.yml@main
    with:
      site_name: wikip.co
      branch: main
      content_ref: ${{ github.event_name == 'repository_dispatch' && github.event.client_payload.content_ref || 'main' }}
      content_sha: ${{ github.event_name == 'repository_dispatch' && github.event.client_payload.content_sha || '' }}
      enable_docker: true
      enable_encryption: false

For anthonyrussano.com (.github/workflows/generator.yml):

Same changes as wikip.co, but with master branch instead of main.

Reasoning:

• repository_dispatch with type content-updated allows external triggers from the content repo
• Passing content_sha through the dispatch payload makes the content selection reproducible
• A reusable workflow keeps the build/publish logic in one place instead of copying shell steps between site repos

Step 9: Add Email Notifications (Optional)

Fetch email credentials from Vault and store as GitHub Secrets:

1
2
3
4
5
6
7
8
9
10
11

# Fetch credentials from on-prem Vault
curl -s -H "X-Vault-Token: <token>" \
  https://vault.wikip.co/v1/secret/data/jarvis/email | jq -r '.data.data'

# Store as GitHub Secrets in both site repos
printf "[email protected]" | gh secret set EMAIL_USERNAME -R wikip-co/wikip.co
printf "<password>" | gh secret set EMAIL_PASSWORD -R wikip-co/wikip.co
printf "smtp.protonmail.ch" | gh secret set EMAIL_SMTP_SERVER -R wikip-co/wikip.co
printf "587" | gh secret set EMAIL_SMTP_PORT -R wikip-co/wikip.co

# Repeat for anthonyrussano.com

Add email notification step to workflows:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18

- name: Send email notification
  if: always()
  uses: dawidd6/action-send-mail@v3
  with:
    server_address: ${{ secrets.EMAIL_SMTP_SERVER }}
    server_port: ${{ secrets.EMAIL_SMTP_PORT }}
    username: ${{ secrets.EMAIL_USERNAME }}
    password: ${{ secrets.EMAIL_PASSWORD }}
    subject: 'wikip.co Build ${{ job.status }}: ${{ github.event.head_commit.message }}'
    to: [email protected]
    from: ${{ secrets.EMAIL_USERNAME }}
    body: |
      Repository: ${{ github.repository }}
      Branch: ${{ github.ref }}
      Commit: ${{ github.sha }}
      Status: ${{ job.status }}
      Workflow: ${{ github.workflow }}
      Run: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}

Reasoning:

• Fetching credentials from Vault maintains a single source of truth
• Storing as GitHub Secrets enables use with GitHub-hosted runners (Vault is only accessible from on-prem runners)
• Using if: always() ensures notifications are sent even if the build fails
• Including run URL in email body provides quick access to build logs

Workflow Execution

When a markdown file is updated in the content repository:

1. Content repo workflow triggers on push to main
2. GitHub Actions sends repository_dispatch events to both site repos, including the exact content SHA
3. Site repos receive the event and trigger their workflows
4. Site workflows:
   • Check out the site repo itself, then initialize public shallowly and the content submodule with full history
   • Resolve the content submodule to the dispatched SHA
   • Restore markdown mtimes from Git history so Hexo updated_option: 'mtime' stays meaningful
   • Build Hexo site
   • Push generated HTML to public submodule
   • Build and push Docker image
   • Send email notification

Common Issues and Solutions

Issue: Rebuild picked up the wrong content revision

Problem: A site rebuild that simply tracks main can drift away from the exact content commit that triggered it.

Solution: Pass content_ref and content_sha through repository_dispatch, then detach the content submodule at that SHA inside the reusable workflow.

Reasoning: The site build becomes deterministic and can be reproduced later.

Issue: Hexo updated dates drift after CI rebuilds

Problem: When Hexo uses updated_option: 'mtime', a fresh checkout makes many files look newly updated unless mtimes are restored from Git history.

Solution: Initialize the content submodule with full history and then restore each markdown file’s mtime from its last modifying commit before running the Hexo build.

Reasoning: This preserves meaningful updated ordering without changing the global permalink strategy.

Issue: public submodule becomes unpopulated after the build

Problem: hexo clean deletes the configured output directory, which in this setup is the public submodule worktree.

Solution: Preserve the public/.git file across the clean/build step so the generated output lands back into the same submodule worktree.

Reasoning: The workflow can clean and rebuild static output without severing the Git identity needed for the publish step.

Issue: Git push rejected during workflow

Problem: Race condition when pushing to site repo while workflow is running.

Solution: This is cosmetic - the important parts (site generation and deployment to public repo) complete successfully. The final push failure doesn’t affect the deployed site.

Issue: Missing TRIGGER_TOKEN secret

Problem: Content workflow fails with “Bad credentials” error.

Solution: Ensure the token is created and stored:

1

gh auth token | gh secret set TRIGGER_TOKEN -R wikip-co/content

Benefits of This Architecture

1. Single Source of Truth: All markdown content exists in one repository
2. Simplified Content Management: Writers only need to commit to one place
3. Automated Deployment: Changes automatically propagate to all sites
4. Version Control: Full Git history for all content changes
5. Separation of Concerns: Content separated from site configuration and themes
6. Flexible Frontends: Each site can have different themes, configurations, or frameworks

Alternative Approaches

Manual Submodule Updates

Instead of automatic triggers, sites could manually update submodules:

1
2
3
4
5
6

cd /home/anthony/wikip.co/site/source/_posts
git pull origin main
cd ../../..
git add site/source/_posts
git commit -m "Update content"
git push

Trade-off: More control but requires manual intervention for every content update.

Monorepo Approach

All content and site configurations in a single repository with separate build paths.

Trade-off: Simpler structure but less flexibility for different site configurations and harder to manage permissions.

Content API

Content served via API (headless CMS) instead of Git submodules.

Trade-off: More dynamic but adds infrastructure complexity and potential latency.

Conclusion

This Git submodule approach provides an optimal balance between automation and simplicity for managing shared content across multiple static sites. It leverages GitHub Actions for orchestration while keeping content selection deterministic through content_sha dispatch payloads and preserving Hexo update dates through Git-based mtime restoration.

The implementation required approximately:

• 3,198 markdown files unified from two repositories
• 3 repositories total (1 content, 2 sites)
• 3 GitHub Actions workflows (1 trigger, 2 deployment)
• 1 Personal Access Token for cross-repository communication

Sources