Securing GitHub Repos After a Supply Chain Attack
This guide walks you through the steps to secure your GitHub repositories after a supply chain attack, such as a compromised dependency or a leaked token. You’ll learn how to assess the damage, revoke access, rotate secrets, and harden your repos to prevent future incidents. Before you start, make sure you have admin access to the affected repositories and a backup of any critical data.
What You’ll Need
- Admin access to the affected GitHub repositories
- Access to your organization’s audit logs (if applicable)
- A list of all integrations, tokens, and deploy keys connected to the repos
- Ability to rotate credentials (e.g., access to your CI/CD provider)
Step-by-Step
- Quarantine the affected repositories. Immediately set the repositories to private or restrict write access. This prevents further malicious commits or exfiltration while you investigate. In GitHub, go to the repo’s Settings > General > Danger Zone > Change visibility, or adjust branch protection rules to require pull request reviews.
- Review the audit log and commit history. Check the Security tab and the audit log (for organizations) to see what actions were taken and by whom. Look for unexpected commits, new collaborators, or changes to branch protection rules. Use
git loglocally to inspect the commit history for suspicious changes. - Revoke all compromised credentials. Go to GitHub Settings > Developer settings > Personal access tokens and revoke any tokens that may have been exposed. Also, check for OAuth apps and GitHub Apps that have access to the repo and remove any you don’t recognize. If you use SSH keys, delete them from the repo’s deploy keys and from your GitHub account.
- Rotate secrets and environment variables. If the attack involved a compromised dependency or CI/CD pipeline, assume that any secrets stored in GitHub Actions secrets, environment variables, or deployment keys are compromised. Rotate them immediately. This includes API keys, database passwords, and signing keys. Update the secrets in the repo’s Settings > Secrets and variables > Actions.
- Audit and remove malicious dependencies. Review your dependency files (e.g.,
package.json,requirements.txt) for any suspicious or unexpected packages. Use tools likenpm auditorpip-auditto check for known vulnerabilities. If you find a malicious dependency, remove it and replace it with a trusted version. Consider pinning dependencies to exact versions to prevent future tampering. - Update branch protection rules. Ensure that your main branches require pull request reviews, status checks, and signed commits. This prevents direct pushes that could introduce malicious code. Go to Settings > Branches > Branch protection rules and add or update rules for your default branch.
- Enable two-factor authentication (2FA) for all contributors. If not already enforced, require 2FA for all members of your organization. This adds an extra layer of security against account takeover. In organization settings, enable the “Require two-factor authentication for everyone in your organization” option.
- Reset any compromised user accounts. If an attacker gained access to a contributor’s account, that user should change their password, revoke all sessions, and rotate any personal access tokens. They should also check their SSH keys and remove any that were added during the attack.
- Clean up any malicious code or commits. If you find malicious commits, you can revert them using
git revertor, if the repo is private and you’re sure, force-push to remove them. Be careful with force-pushing as it rewrites history and can cause issues for other contributors. Prefer reverting to keep a clean audit trail. - Monitor for further suspicious activity. After securing the repo, keep an eye on the audit log and commit activity for a few weeks. Set up alerts for new collaborators, changes to branch protection, or pushes to protected branches.
Common Mistakes
- Only changing passwords, not revoking tokens. Passwords are often not the attack vector; tokens and SSH keys are. Revoke all tokens and keys, not just the ones you think are compromised.
- Forgetting about forks and clones. If the repo was forked or cloned by the attacker, those copies still contain the compromised code. You can’t delete forks you don’t own, but you can contact the fork owners or, if they’re part of your organization, delete them.
- Not rotating secrets that were used in CI/CD. If the attack was via a dependency, the CI/CD pipeline may have been compromised. Rotate all secrets, not just the ones directly in the repo.
- Rushing to delete the repo. Deleting the repo removes the evidence and makes it harder to investigate. Instead, quarantine it and then decide after analysis.
Tips
- Use a dedicated deployment key for each repo. This limits the blast radius if one key is compromised.
- Regularly audit your dependencies. Use Dependabot or similar tools to keep dependencies up to date and alert you to vulnerabilities.
- Enable GitHub’s secret scanning. This can alert you if secrets are accidentally committed to the repo.
- Use signed commits. This ensures that only trusted developers can push code, even if an account is compromised.
- Keep a local backup of your repo. If you need to force-push or revert, having a clean backup can save you.
Troubleshooting
- If you can’t revoke a token because you don’t know which one was compromised: Revoke all tokens and create new ones. It’s better to disrupt your own workflow than to leave a backdoor open.
- If a malicious commit was pushed to a protected branch: You may need to temporarily disable branch protection to revert the commit, then re-enable it. Be sure to do this quickly and with a clear plan.
- If the attacker added a new collaborator with admin rights: Remove them immediately. If you can’t, contact GitHub Support for assistance.
- If you suspect your GitHub account itself is compromised: Change your password, revoke all sessions, and enable 2FA. Then follow the steps above for your repos.
- If the attack was via a dependency that is now removed, but your build still fails: Clear your build cache and reinstall dependencies from the lock file. This ensures you’re not using a cached malicious version.
Frequently Asked Questions
How do I know if my repository was part of a supply chain attack?
Look for unexpected commits, new collaborators, or changes to branch protection. Also, check the audit log for suspicious actions. If you use a dependency that was reported as compromised, assume your repo is affected.
Should I delete the repository and recreate it?
Only as a last resort. Deleting the repo removes the history and evidence. It’s better to quarantine, clean, and restore from a backup if needed.
Can I recover a deleted repository?
If you have admin rights, you can restore a deleted repo within 90 days. Go to Settings > General > Danger Zone > Restore repository.
What if the attacker pushed to my main branch and I can’t revert?
If you have a clean backup, you can force-push to restore the branch. Otherwise, use git revert to undo the malicious commit while keeping the history.
How do I prevent this from happening again?
Enforce 2FA, use branch protection, regularly audit dependencies, and limit the number of people with write access. Also, consider using a dependency lock file and pinning versions.
Do I need to notify my users or customers?
If the attack could have exposed user data or if your software is used by others, you should consider notifying them. Check your legal obligations and be transparent about the incident.
Related
Method: the standard formula for this calculation, rounded as each result states. See how the calculators work or report an error.