The devil is in the details

Implement algorithms that minimize slippage in live forward tests and live trading

Jim Simons, one of the most successful investors ever

The idea

“The group coined a name for the difference between the prices they were getting and the theoretical trades their model made without the pesky costs. They called it The Devil.” Gregory Zuckerman.

The quote above is from the great book The Man Who Solved the Market. In it, Gregory Zuckerman tells the story of Jim Simons, a brilliant mathematician who revolutionized the world of finance with his groundbreaking use of quantitative trading. Simons, who founded Renaissance Technologies, applied complex mathematical models to identify patterns in financial markets, a method that transformed how hedge funds operate. Zuckerman's account explores how Simons, known for his emphasis on precision and data-driven decisions, not only built one of the most successful funds in history but also reshaped the financial landscape by focusing on the importance of minimizing trading costs and maximizing efficiency.

Minimizing trading costs is crucial for any trading strategy, particularly in quantitative finance, where high-frequency trades and small margins are common. Even slight discrepancies between a model's theoretical performance and its real-world execution, often caused by transaction fees, slippage, and market impact, can erode profits significantly over time. This is why Simons and his team at Renaissance Technologies paid close attention to these "pesky costs" and coined the term "The Devil" to describe the gap between the ideal and the actual trades.

A while ago,

Simon M

interviewed Greg Zuckerman on

The Algorithmic Advantage

podcast. It's a great interview:

This week, I will share a code snippet that aims to minimizes slippage while placing orders at the openings in live forward tests. This article will be a follow-up on the Coding live forward tests article, released on Sep-1st. It's going to be another one about coding and with deliberate attention to detail. Let me explain why.

First, I received an overwhelmingly positive reaction from last article. This was a surprise I wasn't expecting. Based on it, I will try to include more texts about logic, showing code snippets that help me on live forward tests and trading.

Second, the strategy I was planning on share didn't quite work. I'm working on sharing new classes of strategies other than mean reversion, as well as new asset classes: next week we will see strategies different than mean reversion/momentum. But that might become a better approach to the weekly articles: I will try to share a new strategy every week; whenever it doesn't quite work, I will share logic and code.

Finally, I've been receiving an increasing number of requests to share my backtesting software. I'm still figuring out the best way to do it. More on that soon :)

This article won't be long, but will help with keeping costs in check - which is crucial while live forward testing and running strategies that trade often. Here's the plan we will follow:

1. First, we will quickly review a strategy that trades often, for which keeping slippage in check is critical to its success;

2. Second, we will briefly describe the problem: how minimizing the slippage while placing orders at market open is challenging;

3. Then, we will show a solution to the problem;

4. Finally, we will share an implementation to that solution.

People say the devil is in the details. Let's see how we can exorcise him.

The chosen strategy

We will use the long & short strategy described in the “Long & Short Mean Reversion Machine Learning” article. I've chosen this strategy because:

• The strategy has a high number of trades: 1,907 trades/year on average, which means 7-8 trades/day;

• The strategy has a strong performance, which is sensitive to the slippage: the backtested annual return might be between 23.1% (20 bps of slippage/trade) and 41.9% (2 bps of slippage/trade).

Impact of slippage and trading costs

Equity curve, drawdown curve, and annual returns for the long & short portfolio with the VIX regime filter

Summary of the trade statistics

Monthly and annual returns

It is a strong strategy, but highly dependent on our ability to automate it and minimize its slippage. Let's see how to do it.

For more details on how the strategy was created, please check the original article.

The problem: placing orders at market open

The strategy described above (as well as most of the strategies I develop) require us to place orders at market open. They assume we can execute the orders at the opening price +/- some basis points to account for slippage and trading costs (depending on the direction of the order).

At first, using Market-on-Open (MOO) orders seem like a great solution to the system requirement. According to Interactive Brokers documentation:

A Market-on-Open (MOO) order combines a market order with the OPG time in force to create an order that is automatically submitted at the market's open and fills at the market price.

So, the only thing we need to do would be running a script that generates and places MOO orders at anytime before 9:30 am US/Eastern, right? Not so fast.

MOO orders have two problems:

• First and foremost, they are not well simulated in a paper trading account, which is critical for us to live forward test a strategy before deploying real capital into it;

• Furthermore, even in live trading, sometimes they do not fill at the official open price; several factors can impact the fill price (the instrument's liquidity, the exchange to which the order is routed, etc.)

How can we address these issues?

A solution

Let's design a simple execution algorithm that solves the problem. First, let's spell out the program requirements:

1. The algorithm must execute orders as close as the opening price as possible, thus minimizing slippage;

2. It's OK for the algorithm to eventually miss an order.

We will start with this simple requirement, then evolve it to ensure it never misses an order.

Before describing the solution, it is also important to understand some aspects about how IBKR API currently works during market openings, which will be useful in the program:

• We can set up IBKR API to continuously stream prices (bid/ask/last) for a list of instruments we provide;

• When we ask IBKR API to stream us prices from an instrument during a trading session, the API will send us the open price once right at the start of the streaming, and then will start streaming bid/ask/last prices continuously as the session goes by;

• If we request IBKR API to stream us prices from an instrument before the market opens, it will start streaming bid/ask/last prices continuously from the pre-market action (as well as the close price from the previous day and a trove of other data), but it won't send us the open price;

• As soon as the market opens at 9:30 am US/Eastern, the API will start streaming the open prices; it will do it only once, as soon as the opening auction finishes;

• The opening auction for each instrument might take milliseconds, seconds, or even minutes, depending on the liquidity of each specific instrument.

The logic

Based on these requirements and our understanding about how the broker's API work, let's describe a simple procedure before translating it into code:

1. The program starts (anytime before the market opens), taking as input the list of orders to be executed;

2. As soon as it starts, our program sets up requests to acquire real time pricing data for all instruments in the list of orders;

3. In the callback method that allow our program to read the prices streamed for each instrument, we instruct it to hold the opening prices in memory as soon clock hits 9:30 am US/Eastern;

4. After starting, our program enters in an infinite loop that continuously check if the opening prices for each of the tracked instruments are already in memory (which might take milliseconds, seconds, or even minutes, depending on each instrument);

5. As soon as each opening price is set, inside the infinite loop iteration, we place a limit order setting the limit price as the open price;

6. Finally, we set a time limit inside the infinite loop, after which we cancel any pending/partially filled orders, break the loop and end the program.

Before translating the logic into code, let's see how it translate into pseudocode.

Pseudocode

The logic described has three main methods that must be implemented:

1. The method to place the orders (the main program);

2. A callback method that listens to the prices being streamed by IBKR API and hold the open prices into memory;

3. A callback method that fires as soon as we get the open/pending/partially filled orders from IBKR API, after a while, and cancels these orders.

Here's the pseudocode for each of these methods:

1. Method to place orders (inputs: list of orders, time limit to cancel)

• Connect to IBKR API

• Loop (only once) though all orders to be executed:

  • For each instrument (aka contract), request the IBKR API to stream its prices

  • Hold the orders to be executed in memory

• Start an infinite loop:

  • Loop through all open prices already set:

    • Check if this order was already placed; if yes, skip; otherwise, continue

    • Set up limit order for this instrument, using the open price

    • Place the order

    • Save this order as already placed

  • Check if time already limit; if yes, request all open/pending/partially filled orders (in this callback, we will cancel them)

2. Callback method that listens to the prices being streamed

• Check current time; if before 9:30 am US/Eastern, return; otherwise, continue

• Check the price received; if it is an open price, save it into memory

3. Callback method fired once we have all open/pending orders

• Loop through all open/pending/partially filled orders:

  • Cancel each of them

That's it. Now, let's see how all this translates into Python code.

Implementation details

Let's check the implementation now. It's important to highlight that I use logs a lot. Why? In my view, this practice significantly improves debugging. While writing code, we spend most of our time debugging, and bugs are an inevitable part of writing code.

"If debugging is the process of removing software bugs, then programming must be the process of putting them in.” Edsger W. Dijkstra. :)

First, let's check the main method:

The code above should be straightforward, especially after seeing the logic and the pseudocode. An interesting and important highlight on the code above are the several data structures used to hold data into memory:

• orders_to_execute and app.contracts_orders, dictionaries that hold the orders to execute using the contracts’ symbols as keys;

• app.symbols_to_track, a dictionary that holds the symbols to stream prices, using the request IDs as keys;

• placed_orders, a list of symbols to control whether an order was already placed, thus ensuring we do not duplicate any order.

The line 149 is extremely important to ensure we will only cancel the unfilled/partially filled orders up until that moment.

Finally, let's check the IBapi class, that implements the interface needed to communicate with IBKR API and receive data from it through the callback methods:

The code above is also straightforward and well documented, so it should be pretty readable even for people unfamiliar with IBKR native Python API. Some interesting highlights:

• Lines 55-59 set data structures to hold important data into memory throughout the program duration;

• Lines 76-79 save all pricing being streamed into a CSV file, which might be handy to debug later;

That's all there is to it. Now, we will see an improved version of the program above which

1. Ensures all orders are executed, no matter what;

2. Provides a good start for the more adventurous readers to implement more complex and interesting logic.

A more interesting version

The first version of our program is my simplest possible implementation to the requirements described earlier. Now that we have it as our baseline, we can build upon it to express more interesting ideas.

Now, let's change our requirements. Let's say that, after a few minutes past the opening, instead of just cancelling the unfilled/partially filled orders, we want to replace them by market orders. How would we express this idea into code?

To accomplish that, we need to add a few lines of code to the openOrderEnd callback method:

The new lines are from 110 to 124. Here's what they do in detail:

• Line 112 change the canceled order from limit to market;

• Line 114 re-place the canceled limit order as market order;

• Lines 106-107 are extremely important to ensure we do not replace orders more than once;

• Lines 119 to 124 deal with an odd edge case: sometimes, the IBKR API take longer to stream the open price or not even stream it at all during the life of our program. In that case, must place the orders directly as market orders.

All the rest is the same.

Final thoughts

There's it! The building blocks shared above complete an important piece of the puzzle on the last article, ensuring we can live forward test strategies that require us to minimize slippage.

On top of that, the logic shared above is a good starting point upon which you can live forward test and live trade more interesting algorithms.

Imagine, for example, that we wanted an execution algorithm to place limit orders on the opening, and move the limit price 1 cent over the open price every 5 minutes until we get everything filled, up to a limit after which the algorithm would cancel the unfilled/partially filled orders. Imagine, in another example, that we wanted an algorithm to move the limit price using a different function (1 cent over the open price after the first 5 minutes, 2 cents after 10 minutes, 4 cents after 15, etc). We can get creative. The code above is a good starting point. Just expand it.

After two articles about execution, next week I intend to resume dissecting new strategies. Hopefully, now in a new class of strategies. There's nothing wrong with mean reversion; it's just that it's good to show ideas in other spaces as well.

I'd love to hear your thoughts about this article. If you have any questions or comments, just reach out via Twitter or email.

Cheers!