nunogrl.com - Infrastructure Security

Enforcing GPG-Signed Commits in Git

2024-03-21T00:00:00+00:00

🚀 Problem & Solution

📌 Context / Backstory

In our collaborative development environment, we discovered that Git's flexibility allows commits under any name and email. This became a security concern when we realized that GitHub identifies users by email, not SSH keys.

sequenceDiagram participant Alice participant Bob participant GitHub Alice->>GitHub: Commit signed with alice@example.com (GPG signed) GitHub->>GitHub: Shows "Verified" commit from Alice Bob->>GitHub: Commit using alice@example.com (no signature) GitHub->>GitHub: Shows commit as from "Alice" (Unverified) Note over GitHub: GitHub matches commits by email, not by SSH or true identity.

⚠️ The Problem

Anyone with repository access can spoof another contributor's identity
Commit history could be manipulated without detection
No cryptographic proof of commit authenticity
GitHub's user identification relies solely on email addresses

💡 The Solution

We implemented mandatory GPG-signed commits, which:

Cryptographically verify commit authenticity
Prevent identity spoofing
Create traceable commit history
Integrate with GitHub's verification system

👥 Who This Helps

Security-conscious development teams
Open-source project maintainers
Regulated environments requiring audit trails
Organizations needing verified commit history

⚙️ Technical Implementation

Let's visualize the GPG signing workflow:

flowchart LR C[Commit] -->|Sign| G[GPG Key] G -->|Verify| GH[GitHub] subgraph "Local System" C G end subgraph "Remote" GH end style GH fill:#f96,stroke:#333 style G fill:#9f6,stroke:#333

1️⃣ Setting Up GPG Keys

# Generate a new GPG key
gpg --full-generate-key

# List your keys
gpg --list-secret-keys --keyid-format=long

2️⃣ Configuring Git

# Configure Git to use your GPG key
$ git config --global user.signingkey <YOUR_KEY_ID>
$ git config --global commit.gpgsign true
$ git config --global gpg.program gpg

3️⃣ GitHub Integration

Export your public GPG key:

$ gpg --armor --export <YOUR_KEY_ID>

Add the key to your GitHub account settings

4️⃣ Enforcing Signed Commits

In GitHub repository settings:

Navigate to Settings > Branches
Add branch protection rule
Enable "Require signed commits"

🛠️ Troubleshooting & Debugging

GPG signing fails: Check gpg-agent configuration
GitHub doesn't show "Verified": Ensure GPG key is added to GitHub
CI/CD issues: Set up proper GNUPGHOME environment
Smart card/YubiKey: Verify proper card reader access

🔁 Optimizations & Best Practices

Use GPG subkeys instead of master keys
Implement regular key rotation
Set up separate signing keys for different contexts
Use environment isolation in CI/CD pipelines
Consider hardware security keys (YubiKey) for key storage

✅ Conclusion & Takeaways

GPG-signed commits provide a robust security layer for Git workflows, ensuring: - Verified commit authenticity - Protected repository history - Clear accountability - Compliance with security best practices

💬 Comments & Next Steps

How do you handle commit verification in your organization? Share your experience or ask questions below!

Password Store with GPG and Git

2024-03-21T00:00:00+00:00

🚀 Problem & Solution

📌 Context / Backstory

We needed a secure way to manage passwords and secrets across multiple servers and team members. Commercial password managers were either too complex, costly, or required external services we wanted to avoid.

⚠️ The Problem

Managing secrets across systems and teams presents several challenges:

Keeping passwords secure yet accessible
Tracking changes and maintaining history
Sharing secrets securely between team members
Avoiding dependency on external services

💡 The Solution

We implemented password-store with GPG encryption and Git integration, providing:

Secure GPG encryption for all secrets
Git-based version control and distribution
Fully local operation with no external dependencies
Command-line interface for automation

👥 Who This Helps

System administrators managing multiple servers
DevOps teams handling shared credentials
Security-conscious users wanting local password management
Teams needing version-controlled secrets

⚙️ Technical Implementation

Let's visualize the password-store workflow:

1️⃣ Generating GPG Keys in Batch Mode

For automated environments:

cat > gpg-server-key.conf <<EOF
%no-protection
Key-Type: default
Subkey-Type: default
Name-Real: Server Automation Key
Name-Email: server@example.com
Expire-Date: 0
%commit
EOF

gpg --batch --generate-key gpg-server-key.conf

2️⃣ Setting Up the Environment

export PASSWORD_STORE_GPG_OPTS="--armor"
export GNUPGHOME=/etc/password-store/.gnupg
export PASSWORD_STORE_DIR=/etc/password-store/store

3️⃣ Initializing the Password Store

mkdir -p "$GNUPGHOME" "$PASSWORD_STORE_DIR"
chmod 700 "$GNUPGHOME" "$PASSWORD_STORE_DIR"
pass init server@example.com

4️⃣ Git Integration

cd "$PASSWORD_STORE_DIR"
git init
git add .
git commit -m "Initial password store"
git remote add origin git@example.com:secrets.git
git push -u origin main

🛠️ Troubleshooting & Debugging

Ensure proper GPG key permissions (700 for directories, 600 for files)
Verify GPG recipient when encryption fails
Check Git remote access rights for sync issues
Monitor Git conflicts when multiple users update simultaneously

🔁 Optimizations & Alternatives

Consider using GPG agent for improved key handling
Implement Git hooks for pre-commit validation
Use Git branches for testing password updates
Consider pass extensions for additional features

✅ Conclusion & Takeaways

Using GPG with password-store provides a flexible, secure, and lightweight method for managing secrets across machines. With Git integration, you get version history, team sharing, and distributed backup—without compromising security.

💬 Comments & Next Steps

How do you manage shared secrets in your infrastructure? Share your experience or ask questions below!

Testing Microservices Securely Using SSH Tunnels

2024-03-21T00:00:00+00:00

🚀 Problem & Solution

📌 Context / Backstory

This is a solution to connect two different servers on different networks.

What I was trying to achieve was to get the server from network-A to reach the port 80 from the network-B.

These networks are different VPCs on AWS, and connecting the networks is not an option, because of some conflicts with DNS.

We needed to test a local microservice using a production endpoint, which wasn't publicly accessible. This endpoint couldn't be mocked or duplicated in staging — and pushing untested code to production would have been reckless.

Additionally, we had a staging microservice that also needed to interact with the same production service. The risk of breaking production due to untested integration was high, but access was restricted for good reason.

⚠️ The Problem

How do we test these microservices — local and staging — against a real production service without:

Exposing the service publicly
Deploying unverified code
Breaking the security model

💡 The Solution

We used SSH tunnels to create secure, temporary links between the testing environments (local and staging) and the production endpoint. This allowed us to route traffic safely without exposing any services over the internet.

👥 Who This Helps

DevOps engineers needing to test services under real conditions
Developers in environments with locked-down production APIs
Homelab or cloud setups requiring secure point-to-point testing

⚙️ Technical Implementation

I can reach both networks from my laptop through VPNs, so the solution I'm sharing here is on how to forward the port 80 from a server to localhost:8080 of the other server.

SERVER=server-net-b
CANARY=server-net-a;

# workstation
ssh -L 8080:$SERVER:80 $SERVER

# workstation
ssh -R 8080:localhost:8080 $CANARY

# CANARY
curl localhost:8080

Here's a diagram of the solution:

flowchart LR subgraph local network L[Local Machine] end subgraph vpc-prod S[Server *webapp-0ef31 localhost:80*] end subgraph vpc-dev P[Canary] end L --- |*webapp-0ef31 localhost:8080*|L L --- |Local Port Forward ssh -L 8080:Server:80| S L -->|Remote Port Forward ssh -R 8080:CANARY:8080| P P --> |*webapp-0ef31 localhost:8080*| P

Let's visualize the different types of SSH tunnels we can use:

1️⃣ Local Port Forwarding (Local to Production)

To test a local service against a production database or API:

flowchart LR L <-->|Local Port Forward ssh -L 5432| S L -->|localhost:5432| L subgraph local network L[Local Machine] end subgraph vpc-dev S[Staging Server localhost:5432] end

ssh -L 5432:localhost:5432 user@prod-host

This allows you to connect to localhost:5432, which transparently tunnels to the production service on prod-host.

2️⃣ Remote Port Forwarding (Expose Local to Remote)

To make your local microservice available to a remote server (like staging or prod):

flowchart LR L <-->|Local Port Forward ssh -R 8080:localhost:3000| S S -->|localhost:8080| S subgraph local network L[Local Machine localhost:3000] end subgraph vpc-dev S[Staging Server] end

ssh -R 8080:localhost:3000 user@remote-server

Now, remote-server:8080 connects to your local microservice running on port 3000.

3️⃣ SOCKS Proxy (Dynamic Tunnel)

To route your web or tool traffic through a secure production jump host:

flowchart LR L -->|web proxy: localhost:1080| L L <-->|ssh -D 1080| J J ==> G subgraph local network L[Local Machine] end subgraph network J[server] G((internet gateway)) end

ssh -D 1080 -C -N user@prod-host

Then configure your browser or curl to use SOCKS proxy at localhost:1080.

With this, you can access any service that is hosted on the production server, such as a database or API.

Also, all your traffic will be encrypted and routed through the production server.

4️⃣ Reverse SSH Tunnel (Access Machines Behind NAT)

To allow remote SSH access into a local machine that's behind NAT.

This is useful when you need to access a machine that is behind a firewall or NAT. In the situation you need to access a machine that is behind a firewall someone else controls, this is a great solution.

the person will connect to the jumbox server and then you will connect to the jumbox server to access the machine that is behind the firewall.

ssh -R 2222:localhost:22 user@jumpbox

Then connect to the local machine from jumpbox:

ssh -p 2222 user@localhost

5️⃣ Persistent Tunnels with autossh

To keep a tunnel alive automatically:

autossh -M 0 -f -N -L 8080:localhost:8080 user@remote-server

🛠️ Troubleshooting & Debugging

Connection hangs? Add -v or -vvv to see what SSH is doing.
Port not forwarding? Make sure GatewayPorts is allowed in sshd config.
Firewall blocking traffic? Test with telnet, nc, or curl to confirm.
Service only binds to 127.0.0.1? Use ssh -L with explicit hostnames.

🔁 Optimizations & Alternatives

For longer-term infrastructure, consider WireGuard or Tailscale for persistent tunnels.
Use SSH certificates to avoid managing multiple authorized keys.
Run autossh as a systemd or runit service for reliability.

✅ Conclusion & Takeaways

Using SSH tunnels allowed us to test services against production safely, without deploying code or violating security constraints. This pattern is lightweight, robust, and fits well into environments where:

VPN is not an option
Public exposure is forbidden
Controlled testing against production is required

💬 Comments & Next Steps

Have you used SSH tunnels in your microservices architecture? Share your experience or ask questions below!