I'm now running Debian GNU/Linux 64bit with 12Gb of RAM and in no way is Bibble/ASP out of memory...
Of course it is out of memory.
No matter how much memory your 64bit box has, a 32bit app is still limited by the 4GB barrier but practically that is closer to 3GB.
While the last release of B5 fixed loads of memory leaks, looks like there is still one active, so it runs out of memory or stumbles across its ownfeet trying to juggle memory allocation (though that is typically a Windows problem).
Cache size is irrelevant for memory usage.
But, having all images cached and not needing preview generation will help memory usage by avoiding preview generation runs that could use up memory