Lorentzian Classification: Machine Learning Driven TradingView Indicator

hey everyone this is Justin and this is another video on a completely free and open source script that I published recently on trading view titled machine learning lorenzian distance classification this script was featured recently as one of the editors pick Publications in the platform and in the past few days it's actually been trending as the number one script on the platform among all the more recently published scripts so because of this it has been getting a lot of questions and requests for a video on this indicator so I took some time to gather up all these comments and messages I've been receiving and I've tried to organize the questions from everyone into various categories that I think would be high impact for this video so I'll try to touch on each of these categories from just a little bit of background and Theory as to how and why this indicator works also I'll be going through some basic optimization of this indicator how to use it on Lower time frames for example and also how to actually back test it using the trading view back testing framework so hopefully along the way I'll also be able to address some of these clarifications as always I will segment out this video according to timestamp so feel free to skip ahead if you're just interested in one of these sections now as far as the actual machine learning algorithm is concerned this is a type of machine learning known as supervised learning and it relies very heavily on label data and the type of supervised learning we're using here is a form of classification as opposed to regression and specifically we're using a nearest neighbors based classification algorithm now the really cool thing about nearest Neighbors in my opinion is that it is literally so dead simple compared to the other types of machine learning algorithms out there you don't need to know any calculus or even linear algebra for that matter or kernel tricks you know a lot of these deep learning methods these days require you to tune dozens of parameters and Hyper parameters just to get something that's halfway decent but nearest neighbors doesn't require any of that in fact it's really just so intuitive you may have even done it by accident like anytime you've gone back and wondered what happened when the RSI was previously 71 and you just looked across history and narrowed in on these handful of points out of really these hundreds and hundreds of points you really only cared about a handful because those in this case are your nearest neighbors for this particular feature Series so that's an example of how you as just the human looking at a chart kind of naturally thinks about data in this fashion in that really all you care about is a certain subset of your historical neighborhood as you call it in order to basically make a conclusion about where price action is going to be going so you may be wondering if for something really so simple as a concept as just finding nearest neighbors why do you even need an official algorithm at all you know isn't this just something you can eyeball and really there's nothing wrong with the approach of eyeballing and as long as it's just one feature series that you're eyeballing it for where it starts to get quite challenging is when you start having other feature Series in addition to this like imagine if we threw a CCI graph up here in an adx and a wave Trend and a wave Trend 3D or another RSI of a different length and you wanted to consider all these things at the same time that would be pretty hard you would really need a more systematic way of determining similarity between two given points so how would you go about doing that well in mathematics another way of measuring the similarity between two points is simply measuring the distance between two points so I've created this distance algorithm viewer to help with visualizing these various distance algorithms and specifically for this indicator I want to focus on the euclidean and laurencian distance algorithms now the euclidean distance algorithm which is what you see here is pretty intuitive for us right so this is the set of all possible historical points and when you just focus on what is closest to us including space it appears to be a sphere and for the most part this works pretty well I mean there's a reason why it's the default a lot of times for these nearest neighbors type algorithms and it actually works pretty well up into the point where you have some anomaly so for example on a Time series a financial time series if all of a sudden you have a major world event happen whether that's a Black Swan or an fomc meeting or an fomc meeting minutes that is enough to really grow the euclidean distance algorithm off and the reason is with enough critical mass this behaves very similarly to how you might expect a very massive object in space-time to behave and that it will actually start to warp the surrounding space time except in this case where we could call it price time because it represents features that are basically describing your price so the standard grid gravity grid as I'll call it would look like this right but the more significant this event is the more buckling and warping you will see of the surrounding fabric of this Continuum this price time continuum and the warping will continue so much that basically space that you would Envision as being far away is actually a lot closer than you might think and suddenly the euclidean distance becomes utterly inadequate I can't describe this very well and the type of neighbors it's giving you are just totally off um and it isn't necessarily representative of of what you would need it under the conditions the Warped conditions of a very significant world event so one way of getting around this however is by changing the distance formula and I did a lot of research into just diving into the literature seeing what type of distance algorithms were being used out there across various domains not just the markets and finance world but just general time series pure analysis right and one thing that really stuck out to me was how robust the lorenzian distance metric was across a wide variety of Time series data sets that was very impressive for me almost always regardless of the time series it almost always outperformed euclidean distance and that's interesting because I suspect that in nature this type of warping effect regardless of whether it's space time or Price time or whatever feature space you're in I suspect that time series tend to have this warping quality about them and I think that this is particularly relevant to the financial time series world so if you look at the actual graphs like this is another indicator that I've built here that that shows you upcoming world events right so these Orange Lines show you fomc minutes for example and fomc minutes are always fascinating because you don't really know what mood the FED will be in that day they could just wake up and just be incredibly hawkish trying to scare people into not getting too euphoric and you know leading up to this the price could just be really ripping and then something is said that just turns it around and you tend to see this General pattern happen quite a bit to varying degrees but you know it's it's definitely there like you can see it happening and in this case euclidean distance is just utterly inadequate like the nearest neighbors that may be you know in this area right here just tell you absolutely nothing about what's going to happen coming up next and to me that's just so cool because I think this is a great opportunity to change distance metrics whenever you get near to a major event like this so what this will look like like essentially is after you have enough Mass added and there's enough buckling of the surrounding fabric of price time in this sense the way you can kind of get around this is just by switching distance metrics altogether so you can see that if you look at the red the red and the orange way out here remember that's supposed to be close to you these are technically closer in laurencian distance than these purples weigh in here or these greens way over here and that's just so weird to think about but it actually has a very profound ramifications so for example if I were to cut down the amount of Neighbors being shown here this is the euclidean distance and then boom that's the laurencian distance the nearest neighbor's algorithm that this indicator is using only requires eight neighbors to cast their vote right so in this one these eight neighbors will all be voting whether price is going to be going up or down for this guy now these ones also get votes but look at who we're who we're asking we're asking basically someone with a completely different RSI value in the past but the the thing is it had a similar CCI and adx value as us and that's always the thing by by compensating with some of these other feature axes you can actually have more flexibility along one of your principal axes like this RSI value so what that allows you to do regardless of how you kind of view it is it allows you to look back in time and as long as CCI and adx and your other features can basically align enough it can give you some incredible amounts of flexibility here and basically it will allow you to consider like entirely new fractals that otherwise wouldn't have been picked up in the euclidean space so it's a very exciting way to combat the whole warping of price time and that's really the main thing I wanted to get across so with that background I will now transition back over to set things and show you how to optimize the indicator and also how to backtest it okay so in this section I'm going to go through the different settings give you an idea hopefully of how to calibrate this indicator regardless of what time frame you're on whether it's a slow time frame or one of the faster time frames the general principle should be the same and then I want to show you how to actually perform a proper back test on this indicator so there are only a few different sections here they're the general settings which correspond to the settings that govern the entire indicator the feature engineering section where you can really get into the fine tuning of different features that you're using for your model you could adjust the feature account Itself by this toggle right here sometimes this can be surprisingly useful I always recommend if you're trying to calibrate to a newer time frame like a fast time frame and you're trying to figure out what features to use or maybe you suspect that there's some better combination that I don't have here I would recommend always starting off at two features and then usually RSI and wave Trend make a pretty dynamic duo but you could even make it even more simple you know less moving parts and just like start off with just a pure RSI indicator so this is the same thing as just being based off of pure RSI and then you can kind of see how other indicators as you gradually expand out to include CCI with a length of 20 how that would affect it so on and so forth so that's the kind of General process I would take always start off small and then move out from there so that's the feature engineering section it can be useful for any time frame the filter section is you can kind of think of it as like your post processing section so I just reset everything back to defaults here but as you can see volatility filter is usually a good one to always have it just prevents some whipsaw from happening especially during these choppy ranging markets the regime filter is an interesting one basically what it will do is make sure that you only enter as the market is transitioning from ranging into a trending market so the more you go towards one 1.0 the less signals you will see because the more strict it is about only entering at certain times so you can see how it took away some signals there but if I go away if I go with the other direction negative one it actually becomes super lenient like it'll allow you to enter during ranging markets trending markets it doesn't care so the happy medium is usually somewhere in between which is where the default is I usually like it a little bit slanted um to negative one just because I like to see options um but if you um want to see less signals especially for you guys in the lower time frames it's important to Jack this value up so that you have less signals polluting your your interface here so that's what the regime filter is typically you want to use regime or adx but not both they they usually don't play well because they're both kind of related to Trend so if you're filtering on the regime it doesn't really make sense to be filtering on adx sometimes but it'll be surprising depending on what time frame you're on how effective adx can be sometimes so it's always worth taking a look at but very rarely would you ever want these together it doesn't make a whole lot of sense and these two are always nice to have I mean this is for ensuring that you're going with the trend so this default is for 200 but you can adjust that as you see fit and that will ensure that you're only shorting in Bear markets and going long and bull markets and especially at lower time frames this is almost a necessity um you don't want to be going Contra Trend a lot if you're trading on like five minutes one minute so I think that was it actually for the filters the kernel settings are interesting because this is like an extension of the filters I would say you can basically disable trade with kernel trade with kernel is kind of like traded with Trend in the sense that it will just try to make its entries in line with the color changes right so the kernel itself a lot of people think it's like a moving average but it's actually a lot better than that you could change the relative weighting for example to be 0.1 and this sometimes gives you a smoother fit overall um you can see it did boost the win rate a little bit over here so that's always an interesting trick that you could try uh of course moving the look back window will also resolve in a tighter fit like you would expect but the regression level is like unparalleled in terms of how tightly you can fit it to a curve and you will really be able to start to see harmonic patterns that you wouldn't be able to see otherwise like over here you can kind of see a head and shoulders pattern forming and that's pretty hard to get if you're just using moving averages maybe if you're using like a a ux moving average or a Kaufman but yeah kernels are pretty awesome you could also slap on a whole another kernel here using my nadaria Watson indicator and this kind of adds like a whole other level of Confluence whenever they cross over you know it's kind of going bearish and vice versa so that's a useful trick that you can do now for actually back testing it this was not meant to be a back testing framework initially but I did add in an option to kind of force this into a I guess a mode that matches more closely the native back tester and the way you can actually get this to work is if you set up a back test adapter here like I have mainly these lines are probably the most important because it's what translates the actual integers over to your start and stop conditions um what you will do is basically load up your back test adapter and then specify your Source by default it will be closed but you want to select back test stream okay and that back to stream will basically open up the whole realm of possibilities of just back testing using trading views native back testing framework and if you are actually using the worst case estimates you can kind of verify that you are in sync with the back tester by going all the way back in time to the start of the 2000 bars in this case it's the 5th of May and you can see here that these numbers are pretty much identical 42 trades 42 trades 59.5 59.52 so that's an interesting way you can just make sure this is actually more or less what you would encounter in the numbers down here and for in terms of just improving this I would recommend almost always you could it just slap on some of these filters down here and like you should for the most part start seeing Improvement and one thing I will also say is the more bars you have historically speaking the better so for example some of these estimates may not be as accurate as some of these estimates and that's simply for the reason that these have a lot more history to kind of consult for selection of their neighbors then then these do so what that means is sometimes to just get more realistic results um you might want to just help your model out a little bit by giving it at least a year to kind of refer back to previous fractals and make predictions off of that so in a nutshell that's how you would back test this that's how I recommend back testing it this was never meant to be a back tester but I did give you access to the um back testing stream so I might publish this if people have trouble making their own but I feel like it's hopefully pretty straightforward so I think that does cover everything I will try to answer as many of the FAQ questions that I couldn't get to in this video over on the trading view section if you have any questions always feel free to reach out and I'll do my best to get back to you thanks