The whole goal of quantisation is to put the data into 'bins' so that it can easily be 'packed' so that you can represent it using less bits (less information). You can think of it like rounding essentially (3.14159 -> 3). Now, sometimes within data, the distribution will be non-ideal for separating it out into bins (let's say that our rounding rules are simple -- we simply use a floor function so 2.45 maps to 2 and 6.4543 maps to 6 etc...) and our bins simply map to the floor -- if we had a set of numbers which look like this: [3.11, 4.43, 5.78, 12.33, 34.32], they would simply map to [3, 4, 5, 12, 34]. Now, we have one huge outlier in our data (34) so to create bins for those sets of numbers, we would need 6 bits of information (2 to the power of 6 = 64), but this is mostly due to the fact that we have one huge outlier (34.32). To get rid of this -- the algorithms applies a random rotation matrix which 'distorts' the original data so that it is more evenly distributed among the possible bins which are assigned to the data set. In linear algebra, a rotation matrix is an orthogonal matrix. When you multiply your vector by this matrix, you aren't changing the "amount" of data (the length of the vector remains the same), but you are recalculating every single number in that vector as a weighted sum of the originals. According to the Central Limit Theorem, when you sum up many random things, the result always starts looking like a bell curve. This is the magic TurboQuant relies on: they don't know what your data looks like, but they know that after the rotation, the data must look like a Beta Distribution and they use this fact to transform the original data into a more 'tightly packed' distribution which allows them to more efficiently pack (or quantise) the information. If most of the transformed data is huddled together into a predictable Bell curve shape, you can pack your bins tightly around that shape leading to much higher precision with fewer needed bits to store it. For example, after applying a rotation matrix, our original transform [3.11, 4.43, 5.78, 12.33, 34.32] might get mapped to something like [8.12, 8.65, 9.25, 10.53, 12.86] and we can crate bins which both are more accurate and need less bits in order to hold our original data set. To create the most optimal bins -- the Lloyd-Max algorithm is used. This algorithm is the gold standard for 1D quantisation. Its goal is to find the best places to put your "boundaries" (where you cut the data) and your "reconstruction values" (the number you store) to minimise the Mean Squared Error (MSE). After applying this, you have your 'rounded' values (or quantized data), but there is still an error value which is missing from our data set: and this is where the residual bit comes in. That bit doesn't represent the original data (or vector) - it simply represents our 'bias' after we apply the above algorithms. It's basically like a '1-bit note' which allows you to perfectly cancel out all the bias terms which our above quantisation algorithm produces to make the 'interactions' (or inner products) when we multiply our values together extremely accurate again even after transforming our original data. Does this make sense?
Amazing explanation! Thank you so much for taking the time to put it together. It makes a lot of sense. I’m not the one who asked the question, but I was impressed by such eloquent and clearly explained answer
This is a fantastic explanation. Thank you. The only part I am not following is how it is guaranteed that 1 bit is sufficient for the error value. Is this something the Lloyd-Max algorithm is responsible for ensuring? (Seems to me that if your quantization algorithm is crappy enough, you could need a large number of bits to store the error.)
Thank you Chase -- I was an early Watsi supporter (and still am actually) but you just reminded me I need to donate soon haha :) Either way fantastic work and thank you!
I believe the MIT license leaves it open for others to host the files for public access. I’d like them to be hosted in one place but others are free to host for their own or business use and/or use them locally. It’s just a way for me to protect my rights as the owner/creator
All are invited to modify the code to suit their needs, but not provide it to the public. If they want to serve it on a local network that is fine. These tools were made to be freely available to all and to modify for their own personal/business needs. If the need arises the latest version will be on Github.
Why in the world would economists need to study this? It's been known that large bureaucracies have been dysfunctional for over a couple of decades now if not centuries. The large reason is because 1) the incentives to do great work are not there (most of the credit for a huge company's success goes to the CEO who gets 100X the salary of a regular worker while delivering usually pretty much nothing) 2) politics usually plays a huge role which gives a huge advantage to your competition (i.e. your competition needs to spend less time on politics and more time on the actual product) and 3) human beings don't functionally work well in groups larger than 100-250 due to the overwhelming complexity of the communication needed in order to make this type of structure work. Incentives though I think are the primary driver - most people at companies like IBM don't have any incentives to actually care about the product they produce and that's the secret behind the ruin of almost every large company.
Edit: you also seem to be giving too much credence to Watson. Watson was actually mostly a marketing tool designed to win in Jeopardy and nothing else. It was constructed specifically to compete in that use-case and was nowhere near to the architecture of a general transformer which is capable of figuring out meta-patterns within language and structurally understanding language. You can read about Watson's design and architecture here if you're curious: https://www.cs.cornell.edu/courses/cs4740/2011sp/papers/AIMa...
More like we need psychologists to ask "why are companies still working with IBM's efficiencies 30 years after its peak?" The workers don't have to care but the businesses dealing with IBM should.
I may be wrong but I think it's mostly for things like enterprise support in case something goes wrong. IBM has had a large footprint in enterprises (WebSphere MQ, etc). People don't want disruptions in case your own kafka cluster with in-house engineers accountable for everything. So having enterprise support for product/ infra gives a sense of safety. At times rightly so. Depends on a lot of factors- risk appetite, capabilities of in-house engineers, what's at stake, and mostly psychological safety, etc.
> most of the credit for a huge company's success goes to the CEO who gets 100X the salary of a regular worker while delivering usually pretty much nothing
Well, in Confluent's case I'm not so sure that's true given that their CEO is also the company founder as well as one of the original authors of Apache Kafka.
Happy Thanksgiving everyone -- I've mostly been a lurker here over the last 20 years and I'm thankful for being able to interact with such a bright and vibrant community full of thinkers, doers and explorers -- you guys definitely changed my life for the better and inspired me in many, many ways.
reply