Okay guys I'm an idiot.
I implemented a new sort/search function which gave me the numbers I posted a couple of days ago. I thought that since it didn't improve memory usage my code must have been solid or I needed to rethink the whole damn thing. But to make a long story short, I forgot to remove all the calls to the old memory hungry code. This caused the former numbers. I'll still be looking into sqlite or similar to see if I can beat the 9999 limit.
Oh and if your wondering it went from 1% of 3gb of ram to 0.4% of said ram. And thats still using the current information caching.
I implemented a new sort/search function which gave me the numbers I posted a couple of days ago. I thought that since it didn't improve memory usage my code must have been solid or I needed to rethink the whole damn thing. But to make a long story short, I forgot to remove all the calls to the old memory hungry code. This caused the former numbers. I'll still be looking into sqlite or similar to see if I can beat the 9999 limit.
Oh and if your wondering it went from 1% of 3gb of ram to 0.4% of said ram. And thats still using the current information caching.