Monitor Internet-Facing Systems for Exploitation Attempts
This guide walks you through setting up monitoring for internet-facing systems to detect exploitation attempts—such as port scans, brute-force logins, and known exploit payloads—so you can respond quickly before a breach occurs. You’ll learn what logs to collect, which tools to use, and how to interpret the signals, starting from a basic setup and moving to more advanced detection.
What You’ll Need
- Access to the servers or network devices you want to monitor (SSH, console, or cloud console).
- Administrative rights on those systems to install agents or configure logging.
- A central log destination: a dedicated Linux server, a cloud storage bucket, or a SIEM (Security Information and Event Management) platform.
- Basic familiarity with the command line and log files (e.g., /var/log/auth.log, /var/log/nginx/access.log).
- Optional but recommended: a firewall or cloud security group that already blocks obvious malicious IPs.
Step-by-Step
- Inventory your internet-facing services. List every IP address and port that accepts connections from the internet. Common examples: SSH (22), HTTP/HTTPS (80/443), RDP (3389), and database ports like PostgreSQL (5432) or MySQL (3306). Note which are essential for business—these are your priority.
- Enable detailed logging on each service. For SSH, ensure the daemon logs to syslog (default on most Linux distributions). For web servers, enable access and error logs—Nginx and Apache do this by default. For cloud load balancers or managed services, turn on access logs in the provider’s console (e.g., S3 bucket for AWS ALB).
- Set up a central log collector. Use a tool like rsyslog or syslog-ng to forward logs from all systems to one server. Alternatively, install an agent like Filebeat or Fluentd on each host to ship logs to a central Elasticsearch cluster or a cloud SIEM. The goal is to have all logs in one place for analysis.
- Install a detection tool. The simplest approach is to use Fail2ban on Linux servers—it watches log files for patterns (e.g., repeated SSH failures) and temporarily bans the offending IP via iptables. For more advanced detection, deploy a Network Intrusion Detection System (NIDS) like Suricata or Snort on a mirror port or a cloud packet capture. For web applications, consider a Web Application Firewall (WAF) like ModSecurity.
- Define what constitutes an “attempt.” Create thresholds: e.g., 5 failed SSH logins within 10 minutes, 100 requests from one IP in 60 seconds, or a request containing a known exploit signature (like SQL injection patterns). Write these as rules in your detection tool.
- Set up alerting. Configure your monitoring system to send notifications when a threshold is crossed. Use email, Slack, PagerDuty, or a simple webhook. Ensure alerts include the source IP, target service, and time window so you can triage quickly.
- Create a response runbook. Decide in advance what to do when an alert fires: block the IP at the firewall, kill the process, take the server offline, or just log and ignore if it’s a false positive. Document the steps and assign owners.
- Test your setup. From a separate machine, run a simple port scan (e.g., nmap) and a few failed SSH logins against your system. Verify that alerts fire and that the response (like IP ban) works. Adjust thresholds if you get too many false positives.
- Review logs regularly. Even with automated alerts, manually inspect logs weekly for patterns your rules might miss—like slow scans or unusual user agents. Use a simple grep or a dashboard like Kibana to visualize trends.
- Keep your detection rules updated. Subscribe to threat intelligence feeds (e.g., emerging threats rules for Suricata) and update your WAF signatures monthly. Attack techniques evolve, so your rules must too.
Common Mistakes
- Only monitoring one service. If you only watch SSH, you’ll miss attacks on your web app or database. Cover all internet-facing ports.
- Ignoring false positives. A rule that triggers constantly will be ignored. Tune thresholds based on your normal traffic.
- Not centralizing logs. If logs are scattered across servers, you can’t correlate an attack that hits multiple hosts.
- Forgetting to test. Many setups fail because the alerting email is misconfigured or the ban rule has a typo. Test regularly.
- Over-blocking. Blocking an entire IP range might take down legitimate customers. Use specific IPs or subnets with caution.
Tips
- Start with Fail2ban for SSH—it’s free, easy, and covers the most common attack vector.
- Use a honeypot (like Cowrie) on a non-production port to attract attackers and see their techniques without risk.
- Enable logging of HTTP headers and POST bodies (with privacy care) to spot SQL injection or XSS attempts.
- Correlate alerts with IP reputation services (like AbuseIPDB) to prioritize responses—a known bad IP is more urgent than a new one.
- Set up a dashboard showing top source IPs, attempted usernames, and blocked events—it helps you spot trends at a glance.
Troubleshooting
- No alerts firing: Check if the detection tool is actually reading the correct log file. Run a manual test and watch the log tail. Ensure the rule syntax is correct (test with a known pattern).
- Alerts too noisy: Increase thresholds or add a whitelist for internal IPs and known scanners like search engine bots. Use a “log only” mode first to see what would trigger, then adjust.
- Logs not arriving at central server: Verify network connectivity and that the syslog port is open (usually 514). Check for firewall rules blocking the traffic. Test with a manual logger command.
- Fail2ban bans the wrong IP: This often happens with proxies or NAT. Configure Fail2ban to use the X-Forwarded-For header for web services, or ignore private IP ranges.
- High CPU usage from monitoring: If Suricata or a log shipper is consuming too much, reduce the number of rules or increase the packet capture buffer. For log shippers, use a smaller batch size.
Frequently Asked Questions
Do I need a SIEM to monitor for exploitation attempts?
No. For small setups, Fail2ban plus a simple log aggregator (like rsyslog) is sufficient. SIEMs are useful for larger environments where you need correlation across many sources, but they add complexity and cost.
What is the difference between a port scan and an exploitation attempt?
A port scan is a probe to see which services are open. An exploitation attempt tries to actually break in by sending malicious payloads (e.g., SQL injection, buffer overflow). Port scans are common and often automated; exploitation attempts are more targeted and serious.
Should I block all IPs that fail SSH login?
Yes, but only after multiple failures (e.g., 5 within 10 minutes) to avoid blocking legitimate users who mistype their password. Fail2ban does this automatically.
How often should I review the logs?
At least weekly for a small environment. If you have high traffic or compliance requirements, daily or real-time alerting is better. Automated alerts handle the urgent stuff; manual review catches the subtle patterns.
Can I monitor cloud services like AWS RDS or Azure SQL?
Yes, but you rely on the provider’s logging. Enable database audit logs and ship them to CloudWatch or Azure Monitor. You can still apply detection rules, but you won’t have direct access to the OS.
What if I find a real exploitation attempt?
Isolate the affected system immediately (disconnect from network), preserve logs, and follow your incident response plan. If you don’t have one, at minimum change all passwords and review for data exfiltration. Consider contacting a professional incident response service.
Related
Method: the standard formula for this calculation, rounded as each result states. See how the calculators work or report an error.