External libraries

[artiko88]

Hey!

What would be your recommendation on using external libraries with the hftbacktest? I tried using scikit-learn while removing @njit decorators, but it severely slowed down calculations. Implementing ML methods like linear regression can be done inside @njit, while others are more complicated :)

[nkaz001]

Since it accounts for full tick data and needs to build the order book and models queue positions, it's really slow without @njit.

It's best to use libraries that are compatible with @njit. You can also call C FFI methods in a jit'ed method. If what you need doesn't depend on a previous action or state of the algorithm and only depends on data, you can pre-calculate and use the pre-calculated value with its timestamp.

[artiko88]

Thank you for your anwer.

Before reading this, i already did the following:

• exported needed features from the hftbacktest simpy via return. Since hftbacktester revolves with while cycle on hbt.elapsed, i simply added features on each revolution to the arrat
• i then calculated some data based on those features and produced another array of the same structure
• then i pass it to the function that starts the backtester as a numpy array (along with instances of Stat and HftBacktest). I am not using precision timestamps, i simply use rows which i extract after each revolution of while cycle on hbt.elapsed.
• Of course, i when i 'import' pre calculated data, i keep everything the same as it was when exporting them

Does this scheme severely impacts precision in a negative way?

[nkaz001]

If you handle everything correctly, particularly regarding the array index so as not to look ahead, it seems fine. However, using timestamps might be safer.