Thursday, March 12, 2009

RAM is not enough - Memory Compression!

This is my first post and what else I could start with!

Memory compression has been my pet project since last 2 years. It adds a compression layer to swap path -- whatever OS swaps out will be compressed and stored in memory itself. This is huge win over swapping to slow hard-disks which are typically used for swapping. Biggest challenges are what to compress? how much to compress? how to manage compressed blocks? handling incompressible data and list goes on...

This is all basically shifting load over to processor to avoid slow disks. Now in multi-core era this seems to only gain in relevance -- de/compression can be almost free if we can use all those cores smartly :)

Project home:
http://code.google.com/p/compcache/

8 comments:

  1. Hi Nitin,

    Nice to know you!

    I have a question, why VMware do not use memory-compression technologies in virtual memory map file (.vmem file)?

    ReplyDelete
  2. Hi,

    I really can't comment on what VMware is thinking in this area. At least for KVM/VirtualBox I can say that the entire guest memory (Windows or Linux, doesn't matter) is compressible with compcache.

    ReplyDelete
  3. Thanks for prompt reply.

    Another question, does VMware plan to support "instant on" vm in future? we know there are several companies are working for "instant on Linux". I am thinking the memory compression technologies should be part of it.

    ReplyDelete
  4. > Another question, does VMware plan to support "instant on" vm in future?

    I don't think so, though I'm not sure.

    ReplyDelete
  5. Ha ha ha...

    It seems that vmware is adopting the memory compression thing....
    http://www.virtualizationpractice.com/blog/?p=6395&utm_source=rss&utm_medium=rss&utm_campaign=vsphere-4-1-released-more-dynamic-resource-load-balancing&lang=pt
    http://vpivot.com/2010/03/01/memory-compression/

    The idea of doing in-memory compression is so old ¬¬ ... I remember the time of QMM in old OSes like Win95/98... but... why the development of such type of thing got freeze?

    I am still trying to find a memory compressor at least for Windows XP, because I need to deal with lot of data on memory... most of it are texts in an in-memory database, also there are some other things [that is only available for Windows] and I really need an in-memory compressor.... any one here knows any software that do it for Windows? Or, there is any chance of compcache be ported to Windows?

    ReplyDelete
  6. > The idea of doing in-memory compression is so old

    Yeah, the idea of memory compression is really ancient. Maybe we didn't have had processors fast enough to handle "real time" memory compression but now we have multi-cores common even on desktops.

    > I am still trying to find a memory compressor at least for Windows XP

    It is possible with compcache, though indirectly:
    http://www.vflare.org/2010/05/compressed-ram-disk-for-windows-virtual.html

    btw, compcache work is completely independent of VMware's work in this area.

    ReplyDelete
  7. "btw, compcache work is completely independent of VMware's work in this area."

    Yes, I know, I just said about vmware for reference to the first question here on this page. :-)

    About compcache, what compression it uses? Well, today CPUs are too fast that I bet that using LZMA2 is fast enough and the compression of it is also the best we can get [as I know LZMA2 is the one used in 7zip, and have the best compression ratio for the compression speed]....
    For example, in a Quad-Core, you can use 2 cores for compression in background works while all other SO operations keep working... I bet it is very transparent to the user, and the gain is very high because the room it opens in memory....

    ReplyDelete
  8. Compcache currently uses LZO compression only. The plan is to support "heavy weight" but better compressing algorithms in future, considering multi-cores (with 2+ GHz) are so common today.

    ReplyDelete