Sayak Paul

I mean we should be able to make the most out of the GPU by reducing the idle-time as much as possible while also ensuring the throughput is really the highest we can get out of the card.

For example, if we are getting 60 QPS, is that the highest we can get out of the card? Is it the maximum limit?

I think we can consider using the cheapest yet reasonable alternative. Okay to probably not exhaustively consider all the specs. For example, it won't make much sense to do this using a 4GB card to do SDXL deployment. So, something in the range of 16-24GB should suffice.

How would you aim for the cheapest latency using existing tooling?

Slick Let's do a project on diffusion models using the cheapest option possible. But we can also show if it can provide the highest efficiency. What say?

So, we replace the FFN layer with FFN layers from different models (which hence requires models to be of the same size).

Crazy that this works!

Haven't gone through the details but a follow-up question.

If the models are needed to be of the same size, how do we select the FFN layers from another model to replace a single FFN layer from the other? If a Transformer block contains a single FFN block (composition of dense layers), how do we accumulate multiple FFN layers, though.

How are the params of the MoE layers populated, though? It doesn't impact the performance? What's the intuition? 😟

Super cool work.

Anyone curious, you can try out Marigold in diffusers through a custom pipeline too. Check it out here: https://github.com/huggingface/diffusers/tree/main/examples/community#marigold-depth-estimation.