Useless is a strong term. I do a fair amount of research on a single 4090. Lots of problems can fit in <32 GB of VRAM. Even my 3060 is good enough to run small scale tests locally.
I’m in CV, and even with enterprise grade hardware, most folks I know are limited to 48GB (A40 and L40S, substantially cheaper and more accessible than A100/H100/H200). My advisor would always say that you should really try to set up a problem where you can iterate in a few days worth of time on a single GPU, and lots of problems are still approachable that way. Of course you’re not going to make the next SOTA VLM on a 5090, but not every problem is that big.
Most CUDA or PyTorch apps can be run through ROCM. Your performance/experience may vary. ZLUDA is also being revived as an alternate route to CUDA compat, as the vast majority of development/intertia is with CUDA.
Vulkan has become a popular “community” GPU agnostic API, all but supplanting OpenCL, even though it’s not built for that at all. Hardware support is just so much better, I suppose.
There are some other efforts trying to take off, like MLIR-based frameworks (with Mojo being a popular example), Apache TVM (with MLC-LLM being a prominent user), XLA or whatever Google is calling it now, but honestly getting away from CUDA is really hard. It doesn’t help that Intel’s unification effort is kinda failing because they keep dropping the ball on the hardware side.
Not really sure who this is for. With soldered RAM is less upgradeable than a regular PC.
AI nerds maybe? Sure got a lot of RAM in there potentially attached to a GPU.
But how capable is that really when compared to a 5090 or similar?
The 5090 is basically useless for AI dev/testing because it only has 32GB. Mind as well get an array of 3090s.
The AI Max is slower and finicky, but it will run things you’d normally need an A100 the price of a car to run.
But that aside, there are tons of workstations apps gated by nothing but VRAM capacity that this will blow open.
Useless is a strong term. I do a fair amount of research on a single 4090. Lots of problems can fit in <32 GB of VRAM. Even my 3060 is good enough to run small scale tests locally.
I’m in CV, and even with enterprise grade hardware, most folks I know are limited to 48GB (A40 and L40S, substantially cheaper and more accessible than A100/H100/H200). My advisor would always say that you should really try to set up a problem where you can iterate in a few days worth of time on a single GPU, and lots of problems are still approachable that way. Of course you’re not going to make the next SOTA VLM on a 5090, but not every problem is that big.
… but only OpenCL workloads, right?
Not exactly. OpenCL as a compute framework is kinda dead.
What types of compute can you run on an AMD GPU today?
Most CUDA or PyTorch apps can be run through ROCM. Your performance/experience may vary. ZLUDA is also being revived as an alternate route to CUDA compat, as the vast majority of development/intertia is with CUDA.
Vulkan has become a popular “community” GPU agnostic API, all but supplanting OpenCL, even though it’s not built for that at all. Hardware support is just so much better, I suppose.
There are some other efforts trying to take off, like MLIR-based frameworks (with Mojo being a popular example), Apache TVM (with MLC-LLM being a prominent user), XLA or whatever Google is calling it now, but honestly getting away from CUDA is really hard. It doesn’t help that Intel’s unification effort is kinda failing because they keep dropping the ball on the hardware side.