Running a big model on a small laptop
High-Performance Custom Laptops – Upgradeable & Powerful
Fully configurable desktop replacement laptops and mobile workstations. Upgradeable, powerful, portable.
Running local models on an M4 with 24GB memory | jola.dev
Experiments with getting usable outputs out of local models on a standard Macbook
Introducing the all new Mecha Comet, live on Kickstarter
Mecha Comet - Open Modular Linux Handheld Computer
Truly yours to own, build and mod. Built and powered by tinkerers just like you.
First look at the DGX Spark
A local supercomputer between the size of a Mac mini and a Mac mini.
New 7-inch Screen Mini Laptop GPD MicroPC 2 Intel® Core™ i3-N300/N250 16GB/512GB Portable Laptop Computer Notebook Windows 11 - AliExpress 7
Smarter Shopping, Better Living! Aliexpress.com
Build Bigger With Small Ai: Running Small Models Locally
Laptops – 3mdeb Sp. z o.o.
LLM in a flash: Efficient Large Language Model Inference with Limited Memory
Large language models (LLMs) are central to modern natural language processing, delivering exceptional performance in various tasks. However, their substantial computational and memory requirements present challenges, especially for devices with limited DRAM capacity. This paper tackles the challenge of efficiently running LLMs that exceed the available DRAM capacity by storing the model parameters in flash memory, but bringing them on demand to DRAM. Our method involves constructing an inference cost model that takes into account the characteristics of flash memory, guiding us to optimize in two critical areas: reducing the volume of data transferred from flash and reading data in larger, more contiguous chunks. Within this hardware-informed framework, we introduce two principal techniques. First, "windowing" strategically reduces data transfer by reusing previously activated neurons, and second, "row-column bundling", tailored to the sequential data access strengths of flash memory, increases the size of data chunks read from flash memory. These methods collectively enable running models up to twice the size of the available DRAM, with a 4-5x and 20-25x increase in inference speed compared to naive loading approaches in CPU and GPU, respectively. Our integration of sparsity awareness, context-adaptive loading, and a hardware-oriented design paves the way for effective inference of LLMs on devices with limited memory.
Mecha Comet - Modular Linux Handheld Computer
A handheld Linux computer powered by open-source software. Modular, programmable, and truly yours to own, build and mod.
Mecha Comet - Modular Linux Handheld Computer
A handheld Linux computer powered by open-source software. Modular, programmable, and truly yours to own, build and mod.
unslothai/notebooks
250+ Fine-tuning & RL Notebooks for text, vision, audio, embedding, TTS models.
SanDisk Extreme PRO with USB4 Review: Fast, Rugged, Huge
SanDisk Extreme PRO USB4 SSD delivers top-tier speed and durability, but its large size may not suit ultra-portable needs.
