A circle for discussing LoRA (Low-Rank Adaptation) and fine-tuning techniques for LLMs.
Created by @taylanturker
Liu, S., Wang, Z., et al.
Interesting improvement over vanilla LoRA. The magnitude/direction decomposition seems to help with learning capacity.
Dettmers, T., Pagnoni, A., Holtzman, A., Zettlemoyer, L.
Game-changer for running large models on consumer hardware. The combination of 4-bit quantization with LoRA is brilliant.
Hu, E., Shen, Y., Wallis, P., et al.
The foundational paper for LoRA. A must-read to understand why this technique has become so popular for efficient fine-tuning.