When you scale orders of magnitude, you have to rethink if exceptions start to make the rules.
I am lucky to come to work (or get on Zoom, as the case is these days) with a set of highly talented people. I love our company and our team because the maniacal focus on craftsmanship and how we do things matters.
Our promise is to bring short term liquidity, heck, for that matter, any financial product at a reduced price, as a fair and transparent exchange for our end user’s data.
This proposition has been very successful, and we are now faced with scaling challenges. Our talented engineers have built highly scalable systems that have made this a relatively easy transition for us. However, there are still growing pains, as with any startup, and we recently experienced what I call an algorithmic scalability issue.
When we started the company in mid to late 2019, Chris, one of our founding engineers, and I would spend countless hours coding and then sometimes drinking out of an empty bar (we worked out of those Spacious offices). Notice I said code and then drink. There were occasions where we would reverse that order, especially when we argued about what constituted our goal of a minimum viable product.
My rough, back-of-the-envelope success metric for our algorithm was an 85% success rate. Fast forward to now. Klover touches the lives of millions of users. That 15% has become a large number, and to keep growing, we have to devise solutions that work for way more than 85% of the users. Fortunately, tackling problem-solving challenges is knitted into the fabric of our engineering culture.
One of my favorite parts of working with the Klover tech team is our solutions sessions. We transparently share the problem we are trying to solve, then we present the approach we took, the solution we came up with, and the limitations of that solution. We invite open discussion during these sessions, and questions come from every corner of the zoom. We’ve even had sessions that led us to rethink our entire solution! These meetings are fun and intellectually stimulating. We work hard to create an environment where every engineer feels eager to share their ideas, questions, and concerns.
The problem that we continually solve is how we can integrate data about our end-user – data from various sources. Some of the nuances that we have to account for include:
· Data comes in coded differently: a simple example is that a company name is represented differently even within the same source over time and certainly could be different across different sources
· The date of a certain activity is not the real date of that activity – again an example here – if I get a bank deposit or withdraw money from the ATM, the actual date of reported activity is different from the actual date of activity since the transactions may have to be “settled”.
· The measure in the data point may not represent the actual measure but post some other transformation – e.g., when you buy something from a store, the amount you pay includes sales taxes and hence is not the exact measure of the price of the product you purchased
These, and other things we haven’t enumerated here, tend to throw off the algorithm we were seeking to develop.
The inspiration for this particular solution came from considerations in the EM algorithm method: https://en.wikipedia.org/wiki/Expectation%E2%80%93maximization_algorithm
When our lead designer sat up, asked many questions about the algo, and even gave me feedback, I took copious notes. If you want to know more about the problem and my solution and want to be part of this great team, send me an email: email@example.com. I am looking forward to chatting with you.