Exchange API Timeout Fail-Safe Workflow
Yes, a fail-safe workflow for exchange API orders that time out or disconnect is achievable, but it requires a deliberate design that treats every order as potentially unfilled until proven otherwise. The core principle is to never assume an order failed just because the API call timed out; instead, you must reconcile the order state before retrying or abandoning it.
Background: Why Timeouts and Disconnects Happen
Exchange APIs are remote services, and network interruptions, exchange-side latency, or maintenance windows can cause a request to be sent but not acknowledged. A timeout means the client stopped waiting for a response, but the exchange may have already processed the order. Disconnects are similar—the connection drops mid-request. The danger is that retrying blindly can create duplicate orders, while giving up can leave you with an unintended position. This is why a robust workflow must include a reconciliation step.
How the Fail-Safe Workflow Works
The fail-safe workflow has three phases: capture, reconcile, and act. When an order is placed, you immediately store the order parameters (symbol, side, quantity, type, price) and a unique client order ID locally. If the API call times out or disconnects, you do not retry the order submission. Instead, you enter a reconciliation phase:
- Query the exchange for the status of the order using the client order ID. Most exchanges support fetching an order by its client ID.
- If the order is found, read its status (filled, partially filled, pending, canceled).
- If the order is not found, it likely never reached the exchange. You can safely resubmit it, but only after confirming it is not present.
After reconciliation, you can decide to cancel, adjust, or leave the order as is. This workflow prevents duplicates and ensures you know your true exposure.
Key Components: Order IDs and Idempotency
The most critical component is a unique client order ID (sometimes called a client order ID or newClientOrderId). This ID is generated by your system and sent with the order. It makes the operation idempotent—meaning the exchange can recognize that a retry is the same order, not a new one. Without it, you cannot safely reconcile. Additionally, you should store the order details locally before sending, so you can reconstruct the request if needed.
Common Problems and Misconceptions
A common misconception is that a timeout means the order failed. In reality, the order may have been filled, partially filled, or still pending. Another problem is that some developers retry the order submission immediately, which can lead to duplicate orders. Also, relying solely on the exchange’s order history endpoint can be slow or rate-limited, so you need a fallback. Finally, do not forget that the exchange may have a different timezone or clock, so always use the exchange’s timestamp for order status, not your local time.
How to Choose a Strategy for Your Use Case
The right strategy depends on your trading frequency and risk tolerance. For a high-frequency bot, you might want a more aggressive reconciliation loop. For a low-frequency manual trader, a simple query-and-confirm approach may suffice. Consider these factors:
- Order type: Market orders fill immediately, so a timeout is more likely to mean the order was executed. Limit orders may rest on the book, so reconciliation is more complex.
- Exchange API capabilities: Check if the exchange supports client order IDs and order status queries. Some APIs have a dedicated endpoint for this.
- Network reliability: If your connection is unstable, you may need a more robust retry mechanism with exponential backoff.
Costs and Risks of Not Having a Fail-Safe
Without a fail-safe, you risk duplicate orders, which can lead to unintended positions and higher fees. You also risk missing a fill, which could result in a missed opportunity or a gap in your hedging strategy. The cost of implementing a fail-safe is mainly development time and the need to handle edge cases. However, the cost of a single duplicate order can be much higher than the development effort.
Alternatives and Advanced Techniques
Some advanced techniques include using WebSocket streams to listen for order updates, which can reduce the need for polling. Another is to implement a state machine that tracks order status transitions. You can also use a database to store order states and a background worker to reconcile pending orders. Some exchanges offer a “cancel on disconnect” flag, but that is not universal and may not be appropriate for all strategies.
Frequently Asked Questions
What should I do if the order status query also times out?
If the reconciliation query times out, you should retry with exponential backoff, but do not resubmit the order. Keep retrying until you get a definitive answer. If the exchange is completely down, you may need to wait and manually review your orders when it returns.
Can I just use a unique order ID to avoid duplicates?
Yes, a unique client order ID is the foundation of a fail-safe workflow. It allows the exchange to recognize a resubmission as the same order, preventing duplicates. However, you must still verify the order status before resubmitting, because the exchange may have rejected the ID if the order already exists.
Is it safe to retry a market order after a timeout?
It is not safe to retry a market order without first checking its status. Market orders execute quickly, so a timeout likely means the order was filled. Retrying could result in a second fill. Always query the order status first.
How do I handle partial fills during a timeout?
If the order is partially filled, you need to decide whether to cancel the remaining quantity or let it continue. The fail-safe workflow should include logic to handle this, such as canceling the remainder and resubmitting a new order for the unfilled quantity, or leaving it as is.
Do all exchanges support client order IDs?
Most major exchanges do, but not all. Check the API documentation for your specific exchange. If not supported, you can generate a unique ID and include it in a custom field, but that may not be as reliable. In that case, you may need to use a different strategy, such as querying recent orders by time.
Should I use a database to store order states?
Yes, storing order states in a database is a good practice. It allows you to recover from crashes and reconcile orders after a disconnect. The database should store the client order ID, order parameters, and current status.
Related
Method: the standard formula for this calculation, rounded as each result states. See how the calculators work or report an error.