Cool idea, but kinda sad that it has to go through a cloud-provider. I feel like there's a possibility with an accelerator-board (Coral TPU or something), to make this into a totally local thing maybe? The longer-waiting time is surely not an issue when considering how many people still use Polaroids.

We were looking to add on-device styles with the Raspberry Pi in order to keep the device cost low, though a Coral TPU would make this easier. The OnyxStream library appears to be able to do SD1.5 generation in 10 minutes on a Pi Zero, so with some optimization and reducing image resolution img2img may be possible on the Pi in ~1 minute. We were also looking at style transfer models, which are much more lightweight and could run fast on a Pi (https://github.com/tyui592/AdaIN_Pytorch/tree/master). Eventually our goal is to make this both on-device and relatively cheap.

We were looking into OnnxStream (https://github.com/vitoplantamura/OnnxStream) and modifying it to support img2img. We got pretty close but yeah capability of running diffusion models on a Raspi are quite limited lol.

Alternatively we could use compute from your iPhone, but it adds additional dependencies to external hardware that I don't quite like. We could use a Jetson, but then power draw is quite high. I agree with you that on-device inference is the holy grail, but figuring out the best approach is something we are still trying to figure out.

why would I believe a random account with 1k followers ?

The fact that I have to give them an email for details just feels immediately like a B2B-scam.

Hope they can figure out software, but what im seeing isn't super-promising

I started using https://github.com/kushalpandya/Petrichor a while ago.

Though design is more akin to the default Apple Music app than WinAmp.

I've wanted a music player like the early versions of iTunes for a while, and this looks like it might fit the bill.

Those who've only known Music.app and later iTunes versions might be surprised to learn that there was a time when iTunes actually had a clean, intuitive UI: https://www.versionmuseum.com/history-of/itunes-app

Beautiful. I remember running iTunes 5 on my Powerbook G4, incredible how things have changed.

Interesting! Will be checking it out.

Always love seeing a new BMC talk

Made my eyes glaze over only slightly faster than trying to listen to the real Zizek.

Well done!

I guess the question is if it's like the crypto-bubble, where theres no real value left in the end (haven't heard of a good use for those ASICs). Or more like the dot-com bubble where fiber-cable installed is still valuable without pets.com around.

But since I wasn't really around for either of those ... ¯\_ (ツ)_/¯

I have no special knowledge but i lean towards more like dot com. AI occasionally does useful things, but nobody really knows what to do with it so people are trying everything and seeing what sticks. Eventually the money will dry up and a lot will fail. However i think chat-gpt-esque things are not going away (the fibre-cable in this metaphor).

Topk sampling with temp = 0 should be pretty much deterministic (ignoring floating-point errors)

> Ignoring floating point errors.

I think you mean non-associativity.

And you can’t ignore that.

Ignoring floating point errors, assuming a perfectly spherical cow, and taking air resistance as zero.

Imagine you are predicting the next token, you have two tokens very close in probability in the distribution, kernel execution is not deterministic because of floating point non-associativity - the token that gets predicted impacts the tokens later in the prediction stream - so it's very consequential which one gets picked.

This isn't some hypothetical - it happens all the time with LLM's - it isn't some freak accident that isn't probable

Okay yes, but would you really say that the main part of non-determinism in LLM-usage stems from this ? No its obviously the topk sampling.

I don't think my tech-lead was trying to suggest the floating-point error/non-associativity was the real source.

> Would you really say that the main part of non-determinism in LLM-usage stems from this

Yes I would because it causes exponential divergence (P(correct) = (1-e)^n) and doesn't have a widely adopted solution. The major labs have very expensive researchers focused on this specific problem.

There is a paper from Thinking Machines from September around Batch Invariant kernels you should read, it's a good primer on this issue of non-determinism in LLM's, you might learn something from it!

Unfortunately the method has quite a lot of overhead, but promising research all the same.

Alright fair enough.

I dont think this is relevant to the main-point, but it's definitely something I wasn't aware of. I would've thought it might have an impact on like O(100)th token in some negligible way, but glad to learn.

Less then 100 people all-in-all.

Shouldn’t be a political play then.

In a good market I’d say you should look around. In this market, keep your head down and get some experience.

I've been at companies as small as 10 and as large as 30,000. and there is no lack of politics in smaller companies from what I've seen.

In my experience, politics can happen at any sized company.

e.g. I've worked at firms with 300,000 people and 150 people. The smaller company had MORE politics in some ways.

Or as a manager of mine in Hong Kong once said:

"In Chinese there is a saying: if there are 3 people then there will be politics"

Fair enough. I guess my take is more that a CEO of a 100 person company should know who the best person for the role is. The CEO of a 100,000 person company is less likely.

I’ve seen what you’ve seen too though: companies that haven’t grown past everyone thinking they report to the CEO turning into gossip factories.

Not a joke. I'm still a student, like half a year away from finishing my masters, so switching jobs at this point feels bit risky/early? Maybe im wrong though

I don't think they meant "quit your job", just "be on the lookout for another".

If your current job is unstable because nobody there knows what they're doing, it's good to have a fallback.

Halve is a verb, half is the noun you were looking for.