Sep 16 2011

Serious Memory Leak 64 Bit Windows 2008 Server R2 SP1

Posted by Mike Brunt at 5:33 AM Web Servers | .NET | DataBase | Languages | Caching | ColdFusion | JRun-J2EE

 

There is a fairly serious memory leak which certainly affects Windows 2008 Server R2 SP1 64-bit on VM Ware ESX 4.1 and ColdFusion 9.01 and which may to be an overall issue with Windows 2008 Server R2 SP1.  It is important to give you the full details as it can be hard to identify and may lead to decisions that it is ColdFusion or the JVM which has the issue and it is decidely not the case; here are the details...

I was on-site at a client and we were load-testing a new infrastructure before going live.  By the way and I realize this is my ongoing mantra, please load test thoroughly before ever moving to new hardware or even more succinctly, before ever putting new code into production.  It is no exaggeration to say that it could save tens of thousands of dollars in lost time and wrong decisions.  So back to my client and I have to give them a shout out, they have a first class team and we were able to work, in harmony, with both the development and infrastructure team to hunt down this issue. That is another key issue and here I am addressing all you VP's and CTO's, it is imperative to ensure all divisions under your watch work cohesively and with understanding of the overall needs of the enterprise; all ego's should be left at the door; this is also critical.  Apologies for the deviation from details here but all of this is truly critical to have any chances of success.

We were load testing fairly stridently and noticed that all VM Ware instances, 64-bit with 8GB RAM (3GB allocated to the JVM Heap, each with two horizontally clustered ColdFusion instances) were almost "maxed-out" on system memory.  We then did what seemed logical and added another 4GB of system RAM to each VM Ware server.  After a restart we noted that once again we were running out of system RAM and looking in Windows Task Manager, it appeared that ColdFusion was taking up 4.5GB of resource when the JVM Heap, was currently set to 3GB maximum, that in my experience had never happened.  So one of the team had the good idea to set down one of the ColdFusion installs to 700MB and once again reboot the VM Ware server, once again and now with 12GB of overall system RAM, memory utilization was climbing and once again ColdFusion appeared to be taking 4.5GB of system RAM, even though it was now set to 700MB maximum.  Another member of the team had been coding away during this time and rolled out some CF Code which was designed to give good details on multiple instance, clustered installs of CF, including heap memory allocated and sure enough, ColdFusion only had just under 700MB allocated.  We then realized we had an overall system problems and sure enough found many articles on the issue.  

This issue was/is not just cosmetic, it impacts performance very markedly so I wanted to get this blog post out there quickly.

 

Comments

Roland Collins

Roland Collins wrote on 09/16/11 9:44 AM

Mike, we run a very similar setup, except on Hyper-V. Server 2008 R2 64bit, SP1, with CF 9.01 64bit Enterprise, and we haven't ever seen an issue like this. We routinely run load tests, and these servers have a very high server load. Our JVM settings are pretty standard, with just a couple of memory tweaks:

-server -Dsun.io.useCanonCaches=false -XX:PermSize=256m -XX:MaxPermSize=512m -XX:+UseParallelGC -Xbatch -Dcoldfusion.rootDir={application.home}/../ -Dcoldfusion.libPath={application.home}/../lib

Minimum heap is set to 1024MB and Maximum is set to 3072.

Just wanted to throw that out there in case it gives you any other information to look at.
Sami Hoda

Sami Hoda wrote on 09/16/11 10:45 AM

We have a similar infrastructure on Hyper-V for internal development and haven't noticed anything like this. Would love to hear more details.
Mike Brunt

Mike Brunt wrote on 09/17/11 3:55 AM

Roland and Sami, thank you for posting your set-up and experiences here. In VMWare ESX there is a capability relating to memory allocation and use called "balooning". As best I could tell, before moving on to present at NCDevcon, these VM's were first created with a memory allocation of 4GB and cloned at that point. This was later increased to 8GB and then 12GB. What we believe is at the time of cloning (at 4GB) some "switch" was set in the ESX server at that level which caused issues and balooning when the memory allocation was increased. However, there are Windows 2008 R2 issues relating to "pre-fetching" of memory in some cases.
pat branley

pat branley wrote on 10/07/11 6:40 PM

Hi Mike

The question is, if we see this kind of leak. how to we fix / prevent it ?
Dawesi

Dawesi wrote on 11/06/11 1:29 PM

Saw a simliar thing on CF8 64 Bit on R2 on an 8Gb Ram instance of VMware....

Same code and setup saw no such issue on Win7 dev
Dawesi

Dawesi wrote on 11/06/11 1:31 PM

might like to add that this was multi-server and two separate instances (not clustered)...
server rentals

server rentals wrote on 11/17/11 9:45 PM

really intersting good one learnt more here
superdry

superdry wrote on 11/18/11 6:12 PM

Every superdry sale fans convinced superdry will be next huge designer. The colors of superdry uk are just amazing because they are universally flattering! It is a great alternative for people who are tired of buying out the entire superdry outlet line and are finished shopping at Our Superdry uk sale store.
qjkider

qjkider wrote on 11/24/11 4:13 PM

Daily meticulously shaping a hair, will make people look extra spirit [url=http://www.bestlisseurghdfr.info/][b]ghd lisseur[/b][/url] Today we introduce some friends of female beauty does not hurt the hair shaping straight hair, [url=http://www.bestlisseurghdfr.info/][b]styler ghd [/b][/url]the product uses a high temperature ceramic plates, [url=http://www.bestlisseurghdfr.info/][b]lisser ghd[/b][/url] hair can minimize the damage will be. Straight hair looks elegant design [url=http://www.bestlisseurghdfr.info/][b]ghd hair straighteners[/b][/url] Body with a high temperature ceramic plates smooth and delicate texture to prevent their hair, [url=http://www.bestlisseurghdfr.info/][b]ghd france[/b][/url] to protect the scalp healthy.
Supra high tops

Supra high tops wrote on 12/06/11 5:02 PM

After read the post,I think this is a good blog.Although i don't understand everything of what your saying, i still enjoyed reading it. Look forward to other subjest post on your
blog,thanks.
Supra TK

Supra TK wrote on 12/06/11 5:02 PM

This is my time to visit here. I found so many interesting stuff in your blog especially its discussion.Thank you for your sharing,I think you must be a funny person in daily life.
Supra high tops

Supra high tops wrote on 12/06/11 5:02 PM

I have read the same idea at other articals,but your post give me a different feeling,and give me some touch,that felling is so good.I will save your website,and waiting for your next post.Hope common progress.Thanks.
Supra trainers

Supra trainers wrote on 12/06/11 5:02 PM

Thanks for such a great post and the review
Kishore Karavadi

Kishore Karavadi wrote on 12/23/11 8:18 AM

Hi Mike, I am also facing the same issue. Do you have any solution for the same?

Thanks
gt98us

gt98us wrote on 12/30/11 10:19 AM

Mike, we too are having issues with 2008 R2 and RAM allocation. First I am a system admin and not a developer. This first peaked it's head with a cloned VM Guest on Esx 3.5 with 8 GB's of RAM. After 22 hours of running all 8 GB of RAM was allocated and the system performance was seriously degraded. Now I am looking at physical servers as well and I am seeing a trend on 2008 64bit and 2008 R2 servers and how they allocate RAM with servers that have large amounts of RAM.

I believe it's tied to the pagefile and physical RAM. I ended up removing the pagefile altogether from the VM guest and the system is performing very well. I would suggest using MS tool "rammap" and it will show you how the RAM is being utilized. Then it starts the debate on whether we need a pagefile at all. MS Kb's mention you technically don't even need one with a 64 bit server that has a large amount of RAM as long as you don't need a memory dump.
hope this helps, I am still researching myself.
Ios On Android Phone

Ios On Android Phone wrote on 01/18/12 9:07 PM

It is a great alternative for people who are tired of buying out the entire superdry outlet line and are finished shopping at Our Superdry uk sale store.
Chad

Chad wrote on 04/27/12 9:20 AM

Any idea if this issue was fixed with VMWare 5?
Mike

Mike wrote on 05/23/12 8:40 PM

Experiencing the exact same problem and running VMWare 5.0.0. Any updates?
rak toko swalayan murah

rak toko swalayan murah wrote on 06/17/12 10:03 AM

This posting is very cool and interesting post.This is a great inspiring article.I am very much pleased with your good work.You put really very helpful information in this site.

http://www.rakminimarket.com

therefore, I would like to thank you for the efforts you have made in writing this article.
Claes

Claes wrote on 09/13/12 6:08 AM

Hi guys.
Discussing the mem leakage,, i can link one fix that is "trial" from Microsoft

http://support.microsoft.com/kb/981314

Should close the mem leak
agence référencement web

agence référencement web wrote on 10/29/12 3:12 PM

Un grand bravo pour la conception de votre blog. Je vous remercie une fois de plus. Amicalement,

référencer son site wordpress http://googlepenguinfrance.wordpress.com référencement d'un site web sur google
Ivo

Ivo wrote on 11/13/12 2:15 AM

Hello,
Like Mike Brunt said this is probably related to VMware ballooning.
Edit your VM and check under the resources tab and make sure the checkbox unlimited is checked.
Otherwise your VM will probably be limited to the amount of memory you´ve originally assigned during creation.

The limit can be on the VM or in the resource pool but most of the time it´s a VM limit
Ivo

Ivo wrote on 11/13/12 2:16 AM

oh, forgot to mention.
To activate the new unlimited setting you have to reboot the VM or vMotion it
Cliff

Cliff wrote on 11/21/12 9:03 AM

I've been having problems with windows 2008 R2 + CF 9 - sites hanging due to CF heap. I found the solution here: http://gruchalski.com/2011/03/24/coldfusion-9-crashing-try-different-garbage-collection/
technical writing

technical writing wrote on 04/26/13 2:19 PM

I think IT is the future on business and industry. We do about everything on internet, we buy we sel we... Internet is now our friend for ever. He is friendly and good for everything.
Las Vegas

Las Vegas wrote on 06/04/13 12:56 PM

ColdFusion was originally created by the Allaire Corporation. In 2001, thus Allaire Cold Fusion became Macromedia Cold Fusion . Adobe acquired Macromedia in 2005 and is still actively developing ColdFusion.
bookshops

bookshops wrote on 07/10/13 10:24 AM

Windows Server 2012 powers many of the world's largest datacenters, and delivers value to organizations of all sizes in between, all around the world.
cristiano ronaldo mercurial

cristiano ronaldo mercurial wrote on 10/28/13 5:02 AM

Your blog post is undoubtedly exquisite thank you very much for one's reveal.You could also to see our online business maybe currently have everything you prefer.
new nike mercurial 2013

new nike mercurial 2013 wrote on 11/21/13 1:10 AM

It’s nice.I truly appreciate your blog,you have carried out the fantastic work.
legoindy2game.eu

legoindy2game.eu wrote on 02/02/14 11:06 PM

I will make sure to bookmark your blog and will eventually come back someday. I want to encourage you to definitely continue your great posts, have a nice holiday weekend!
games forge

games forge wrote on 02/02/14 11:06 PM

Great website, continue with the good work!
oyster card

oyster card wrote on 02/02/14 11:07 PM

Very nice post. I simply stumbled upon your weblog and wished to mention that I've really enjoyed browsing your weblog posts. After all I'll be subscribing for your RSS feed and I hope you write once more soon!
replica watches factory

replica watches factory wrote on 03/16/14 11:41 PM

After a short period of time we manage to clam the ladder and be the #1 site to offer Haitian <strong> replica Franck Muller watches </strong> video totally unbiased . If you get a salad, main course, dessert and <strong> replica swiss watches </strong> after, expect to pay 150 EEK; for just a main course, 80 EEK.
sparxx rx

sparxx rx wrote on 10/16/14 8:52 AM

Perhaps you and your partner are looking to ease into the sex toy world with a small vibrator, or maybe you’re a women looking for a large and in charge makeshift dick when you are home alone.
royall construction

royall construction wrote on 11/02/14 12:48 AM

The combination of uncompromising quality of construction and innovative design make the Audemars Piguet Royal Oak watch a timepiece to wear with pride.
entertainment-music

entertainment-music wrote on 11/02/14 12:48 AM

Music is one of the most universal languages known to man. Many cultures have distinctive styles and sounds of music which can now be enjoyed by everyone, thanks to the products such as recording devices.
www.smartgrideducation.org

www.smartgrideducation.org wrote on 11/02/14 12:48 AM

A recent study conducted by Market Strategies International, involving 1,168 participants from across the US, shows that smart grid and smart meters are still lesser known concepts in the country.
the home stop

the home stop wrote on 11/19/14 6:53 AM

Will be your home toxic-free? Significantly less this content claims. Moreover, toxins will be in an amount that could be hazardess to the health of your self.
counseling education

counseling education wrote on 11/19/14 6:53 AM

Informative training packages allow students to sign up in both in their free time and fulltime degree packages. Students can purchase a bachelors, masters, or doctorates degree in neuro-scientific counseling.
ormiston education

ormiston education wrote on 11/19/14 6:54 AM

Education is good for all. People need education and learning for create themselves pertaining to future faith. So, nothing the real reason for takes some dough from men and women via education and learning.
www.scorespro.com

www.scorespro.com wrote on 01/26/15 11:41 PM

As being a soccer mom is the same as having the part-time job having a demanding routine. My top ten soccer mother ideas may hopefully assist you in finding balance that you experienced and simultaneously help improve your son or daughter's soccer abilities.
ristorante monopoli

ristorante monopoli wrote on 04/03/15 11:18 PM

Many entrepreneurs are thinking about putting up their own restaurants. Restaurants are generally seen as good business ventures because people are always looking for food.
clothing manufacturer

clothing manufacturer wrote on 07/04/15 4:22 AM

Hi Mike

The question is, if we see this kind of leak. how to we fix / prevent it ?
Android Monitorng Review

Android Monitorng Review wrote on 09/15/15 12:31 PM

This is very informative post I didn't know about this but now there is very things that help me in future.
シャネル激安

シャネル激安 wrote on 12/02/15 10:43 PM

シャネルの財布、腕時計、靴、眼鏡、ピアス、バッグかばんが使うの皮質が材料を作るのため、
そのため液体を採用してそれに対して拭くことができます。しかし、
説明したのが必要だのは、使う液体の別に普通でない水、
シャネル激安http://www.cctoh.com/
シャネルバッグ
シャネル財布
シャネル腕時計
シャネル靴
シャネル眼鏡
シャネルピアス
シャネルバッグ新作
シャネル専門のお手入の液体で、かばんの上の現れるあかに対してとても強い的確さがあって。
シャネルバッグ新作 本入手困難ラムスキンセカンドバッグ8016ゴールド金具
boom beach trucos

boom beach trucos wrote on 05/14/16 5:55 AM

I just stumbled upon your blog and wanted to say that I have really enjoyed reading your article. I’ll be subscribing to your feed and I hope you will post again soon
thesecretofstyle

thesecretofstyle wrote on 05/23/16 6:25 AM

A corporation doesn’t exist within a vacuum. Many people-a firm, hospital, charitable group, or govt agency-is just shaped over the society in which it ended up being made. Your ideals, ways linked to thinking, and customs in the culture, amidst other aspects, are reflected from the structure and also behavior linked to organizations on the inside that life style.
fashionablysocial

fashionablysocial wrote on 05/23/16 6:25 AM

Diamonds will almost always be in structure and fashion. Signifying female’s wonders, your female feels full after boosting diamond accessories. Diamonds incorporate extra splendor to just about any material including gold, silver or maybe platinum in which they usually are fixed. Diamond’s shimmer is unmatchable in fact it is aura own always utilized women.
fashiondelights

fashiondelights wrote on 05/23/16 6:26 AM

In relation to the coolest in girls’ fashions for ages 2 for you to 6, mix along with match is otherwise engaged and synchronized looks will be in. This implies girls who will be styling contain the bases coated from go to toe.
cheats for csr racing

cheats for csr racing wrote on 05/25/16 5:33 PM

I was searching the blogs for a long time and that is good to know that I have found such blogs here. Great working! Keep it up!
مواقيت الصلاة

مواقيت الصلاة wrote on 07/19/16 11:23 PM

Diamonds will almost always be in structure and fashion. Signifying female’s wonders, your female feels full after boosting diamond accessories. Diamonds incorporate extra splendor to just about any material including gold, silver or maybe platinum in which they usually are fixed. Diamond’s shimmer is unmatchable in fact it is aura own always utilized women.
Time Attendance

Time Attendance wrote on 07/28/16 9:27 AM

I am so dazzled to be here and to examine this perfect post ever. A commitment of thankfulness is all together for offer this with us.Glad to investigate this perfect post and to accumulate such edifying stuff.
sararobin1910

sararobin1910 wrote on 08/07/16 10:15 AM

This is really a marvelous read for me. I have bookmarked it and I am foreseeing examining new articles. Continue doing magnificent!. <a href="http://www.wipaq.ae/biometric-time-attendance-system-dubai">Biometric Time Attendance System Dubai</a>
Time Attendance System Dubai

Time Attendance System Dubai wrote on 08/07/16 10:16 AM

This is really a marvelous read for me. I have bookmarked it and I am foreseeing examining new articles. Continue doing magnificent!.
SantosKip

SantosKip wrote on 09/12/16 3:26 PM

It is necessary to have first-aid kit for your pets. To discover and trace a phone number it is crucial to choose a reliable website. Ask your bank when can place a password on your accounts.
http://youtubeso.ml/video/d65867af1544c2a753cba90e9ac12cdb-4476.html