Is there any difference for my purposes between live and installed Ubuntu?
Not much. You're going to have lower performance because you're running off a USB stick or DVD and so it'll take longer to load things. You'll also not have as much RAM available because you've got much of the system loaded into RAM.xc2xa0
If you're trying to compile software & other CPU intensive things I wouldn't suggest doing them on the LiveUSB/LiveDVD unless you've got 4GB or more of RAM.
I understand I cannot store any files, but that's fine.
Actually, you can! If you use UNetBootin (which can be downloaded & run on all platforms (Windows, OS X, & Linux) you can tell it to create space on the USB stick to store changes between reboots.xc2xa0
Granted, you'll need something larger than 4GB (8GB+), but you can actually store stuff between LiveUSB boots. Note, this is only for Ubuntu distros:
You can also do something similar in Startup Disk Creator, but you'd need another USB stick to install the ISO on, since you obviously can't write the ISO to the USB stick you're running off of: