ThisCalculator

Everyday

Detect Unusual API Usage for Model Extraction

Updated · Standard formula, shown below

This guide helps you spot API usage patterns that may indicate someone is trying to extract your machine learning model by querying it repeatedly. You’ll learn what signals to look for, how to set up monitoring, and what to do when you find suspicious activity. You’ll need access to your API logs and a basic understanding of HTTP requests and response codes.

What You’ll Need

  • Access to API request logs (e.g., from your API gateway, load balancer, or application server).
  • A log analysis tool (even a simple script or spreadsheet works).
  • Knowledge of your API’s normal usage patterns (baseline traffic).
  • Ability to set up alerts or notifications (e.g., email, Slack, or a monitoring service).

Step-by-Step

  1. Collect and centralize your API logs. Ensure all requests to your model endpoint are logged with at least: timestamp, IP address, user agent, request parameters, response code, and response time. If you use a cloud provider, enable access logging on the API gateway or load balancer.
  2. Establish a baseline. Analyze a week or more of normal traffic to understand typical request rates, the number of unique IPs, average request size, and common query patterns. This baseline will help you spot deviations.
  3. Monitor request volume per IP. Aggregate requests by IP address over short time windows (e.g., per minute or per hour). Flag IPs that exceed a threshold like 10x the average per-IP rate. A single IP making thousands of requests in a few minutes is a classic sign of automated extraction.
  4. Check for sequential or exhaustive queries. Look for requests that systematically vary one input parameter while holding others constant. For example, if your model takes a numeric feature, a series of requests with values 0.01, 0.02, 0.03… suggests someone is probing the decision boundary. Similarly, many requests with slightly different combinations of inputs may indicate an attempt to map the model’s output space.
  5. Examine the distribution of input values. Compare the distribution of input values in recent requests to your historical baseline. A sudden spike in rare or extreme values (e.g., very high or low numbers) could be an attempt to find the model’s limits.
  6. Look at response times. If your model’s inference time is consistent, a sudden increase in response time variance might indicate that the requester is sending more complex or adversarial inputs. Also, extremely fast requests (e.g., < 10ms) could be a sign of a script that is not actually processing responses, just hammering the endpoint.
  7. Check for unusual user agents. Many extraction scripts use default Python or curl user agents. If you see a user agent like “python-requests” or “Go-http-client” making a high volume of requests, that’s suspicious. Also watch for user agents that change frequently from the same IP.
  8. Monitor for high error rates. A high rate of 400 or 500 errors from a single IP might indicate a script that is trying to guess valid parameters or that is not handling responses correctly. However, be careful: some extraction attempts are designed to avoid errors by sending valid requests.
  9. Set up alerts. Based on your thresholds, configure alerts for when an IP exceeds the normal rate, makes an unusual number of sequential queries, or triggers other flags. Use your monitoring tool or write a simple script to send notifications.
  10. Investigate flagged IPs. When an alert fires, check the IP against threat intelligence databases or look up the organization that owns it. If the IP belongs to a known cloud provider or proxy service, it’s more likely to be an automated attack.
  11. Take action. If you confirm malicious extraction, you can rate-limit the IP, block it, or require API keys and authentication. For high-value models, consider adding CAPTCHAs or requiring user accounts for access.

Common Mistakes

  • Setting thresholds too low. You’ll get flooded with false positives. Start with a high threshold and adjust based on your baseline.
  • Ignoring the baseline. Without a baseline, you can’t tell what’s “unusual.” Always compare to historical data.
  • Only monitoring request volume. Volume is important, but sequential queries or unusual input distributions can be just as telling. Use multiple signals.
  • Forgetting to log all relevant data. If you don’t log request parameters, you can’t analyze input patterns. Ensure your logs include the full request body or at least the key parameters.
  • Blocking IPs too aggressively. Some legitimate users may share an IP (e.g., corporate NAT). Use rate limiting rather than permanent blocks when possible.

Tips

  • Use a dedicated monitoring tool. Tools like Grafana, Kibana, or even a simple Python script can help you visualize and analyze logs.
  • Track the number of unique inputs per IP. If an IP sends 10,000 requests but only 100 unique input combinations, it’s likely extracting the model rather than using it normally.
  • Look for patterns in time. Extraction often happens in bursts, e.g., overnight or on weekends when you’re less likely to notice.
  • Consider adding a small amount of noise to your model’s outputs. This can make extracted models less accurate without harming legitimate users.
  • Use API keys even for public demos. Requiring a simple API key makes it easier to track and throttle usage.

Troubleshooting

  • You’re getting too many false positives. Adjust your thresholds upward, or require multiple signals (e.g., high volume AND sequential queries) before alerting.
  • You suspect extraction but can’t see it in the logs. Check if you’re logging the full request body. If not, enable it temporarily. Also check if your load balancer is stripping headers like user agent.
  • A legitimate user is being blocked. Set up a whitelist for known good IPs, or use rate limiting that allows bursts but caps total requests per minute.
  • Your monitoring script is too slow. Use a log aggregation service or stream logs to a fast data store like Elasticsearch to analyze in near real-time.
  • You can’t tell if an IP is malicious. Use WHOIS or an IP reputation service to see if the IP is associated with a cloud provider or known abuse.

Frequently Asked Questions

What is model extraction?

Model extraction is a type of attack where someone queries a machine learning model with many carefully chosen inputs and uses the responses to train a copy of the model, effectively stealing its functionality.

How quickly can an attacker extract my model?

It depends on the model’s complexity and the number of parameters. A simple model might be extracted with a few thousand queries, while a large deep learning model could require millions. The faster your API responds, the faster an attacker can extract it.

Can I prevent model extraction entirely?

Not completely, but you can make it much harder. Rate limiting, requiring authentication, and adding noise to outputs are effective deterrents. There is no foolproof defense, but you can raise the cost of extraction.

Is it legal to block an IP that I suspect is extracting my model?

Yes, you can block or rate-limit any user of your API, as long as you’re not violating any contractual obligations. It’s your service, and you can set terms of use.

How do I set up a simple alert script?

You can write a Python script that reads your logs (e.g., from a file or a database), counts requests per IP, and sends an email or webhook when a threshold is exceeded. Many monitoring tools also have built-in alerting.

Should I hide my model’s confidence scores?

Confidence scores can be useful for extraction because they reveal more information about the model’s internal probabilities. If you don’t need them for your product, consider omitting them from the API response.

Related

Method: the standard formula for this calculation, rounded as each result states. See how the calculators work or report an error.