My Personal Hell = Bukkit Memory Leak

Hi,

Over the past few days I have been dealing with a memory leak that leads to a CPU lock up on my server:
Its specs are as follows
Windows Server 2008 r2
AMD phenom 1055t x5 2.8 ghz
16 gb of ram --> 10 gb dedicated to craft bukkit
100 mbit dedicated connection

The plugins that I was running when the problem initially occoured, and it consumed over the specified amount of ram I allocated (peaked at 12gb) while consuming all cores at 100% processing power (complete lock up)
Essentials 2.9 with geoip and xmpp
Group Manager
Core Protect
Raw critics Ore obfusticator
Anti Cheat
Anti Logger
Plugin metrics
Smoke TP
Votifier
World edit
World Guard
Gravity Sucks
Lift
Iconomy
No-lagg
Simply Vanish
Backup

After initially encountering the problem I removed both the ore obfusticator and no lag from my plugins roster. The memory leaking was less severe, but the problem returned with memory leaks at the 6gb level (I was normally seeing 2gb average so were still triple) and processor consumption at the 90-95% utilization level.

So I backed up my Plugins library, did a dry run, no plugins, fresh CB re-install

Memory leaks are still in the 5gb range for craft bukkit alone, and processor utilization is still at the 90% level. Any pointers?

EDIT: I'v been using CB recomended build 1.2.5-r4 b2222jnks

 

I've had some bad memory leaks involving NoLagg and ore obfuscators recently. Might want to try experimenting with those.

 

I did remove those, but the leaks persist. Raw critics ore obfusticator was the plugin that made the first round of leaks start, as I accidentally disabled it installing No-lag which after re-enabling it the leaks just peaked out and made me sercon into the server to kill java. My problem is now even after uninstalling those plugins my leaks persist

 

I'm not quite sure how you do it on a windows server, but if you can obtain a heap dump from the server process, you can analyze it for leak suspects in Memory Analyzer Tool. Usually the involved plugin will have its name in the details.

 

I Have dumped the process once and I am inspecting the dump file, but still I'd rather get help fast rather than sit and do it the long way....

 

Steer clear of NoLagg, that seems to be a major issue with the current build of Bukkit. One of my developers also mentioned that it's critical to ensure that you (or your host) has the most updated version of Java, and a specific build of Java, though I cannot remember it. Send me a PM and I might be able to assist you further via Skype.

 

I have JRE 7 U5 if that helps, but the problem is persistent after the plugin removal, which makes me think something has ghosted. And I have no clue where my temp files are hiding, I looked in %appdata%...

Also I am the host

 

Go into SSH and type:
java -version

Post the whole response here.

 

Ill just pull it from the files properties

7.0.50.5

 

What version of those plugins are you running?
An older build of CoreProtect will do that, but that has been resolved in the last release or two.

Have you ensured all your plugins are up to date? Have you removed them one by one to find the problem?

Also, its good to note that a single misbehaving plugin may not be causing you problems. However, if two or three misbehaving plugins are running at the same time, they could be compounding the issue and taking you down.

What is the line you use to start your server? Do you have any server.log files to share?

 

Well I wish my log wasn't 15mb of text or I would really share it, but removing the obfusticator and NO-lag (which seemed to be the compounding two) stoped the Super leak I was seeing, but my server just sitting with the same list of plugins as above all up to date (checked that already I try to stay on top of that XD), is still just idily eating memory.
I started it an hour ago, an from the logs nobody has connected yet, minus some status pinger's, and it has gone from 1gb on the start to now 3.5gb, which leads me to believe that there is a static leak. If you can wait 15 minutes I pull up a java thread dump for you. I also have a windows static memory dump, but there is no way I'm going to be able to share 11gb of memory data on here...

Here are the logs that I provided to the plugin maker for raw critics ore obfuscator.
http://pastebin.com/qHyc1gjY

 

What are your runtime GC options? I use these, and never have memory issues that are do to anything besides maps, players, or plugins. In fact, my memory usage when idling stays at about 25% (out of 2GB's)

Code:

-Xmx{MAX_MEMORY}M -Xms100M -server -XX:+DisableExplicitGC -XX:+CMSParallelRemarkEnabled -XX:+AggressiveOpts -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:ParallelGCThreads=4 -XX:SurvivorRatio=19 -XX:TargetSurvivorRatio=90 -XX:MaxGCPauseMillis=500 -jar craftbukkit.jar nogui

 

10gb is the max

but here is the look of my windows run.bat

Code:

 "C:@ECHO OFF
SET BINDIR=%~dp0
CD /D "%BINDIR%"
"%ProgramFiles%\Java\jre7\bin\java.exe" -Xmx10G -Xms1G -jar craftbukkit-1.2.5-R4.0.jar
PAUSE
 
 

 

rossfudgew
Your server.log is not being rotated if its 15MB. You only need to pastebin the parts from the last start to finish of the server.

What makes you think you are using 5GB of RAM on a fresh startup, no plugins and a new world?

 

Um the data sitting infront of me, as in I have a remote desktop connection and I see Java.exe using 5gb of memory, but since you don't beleive me here's an image of my last SERCON session from when I had to manually crash java



And if you can't read that data I'll do it for you here

Java.exe had a total processor time of 2:33:18.052
WS = 3960348
Faults = 1632018
commit (ram) = 5136092 (Hmpth 5gb)
priority = 8
handles = 167093 (now why the hell does bukkit need that many handles?)
threads = 19906 (better yet why does bukkit need so many threads?)
pid= 9948 (note the kill below)
name = java.ex (ran out of room for exe)

 

Can you provide the server.log for the time you ran the server with no plugins and with it generating a new world?

 

I figured out the problem anyways. Where CB is stored in my server, the directories permission got corrupted because of the crashe(s) (sercon forced crashes can mess with ya), which then caused the server to not initialize properly. I fixed it by deleting the folder and starting completely over with the CB install, then I imported my backups