High cpu usage on bukkit 1240. Can't locate the cause.

Welcome to my really long thread of what I'd tried and learned about my problem. Note to those that post something to try that is already listed below I'm going to silently thank you for the free bump. I see too many canned responses on this forum of do <X> when <X> was listed as one of the first two things tried. Not saying this to discourage replies but to hopefully limit the one line posts of "Try running the server without plugins", "Try a fresh map", etc. We've all seen plenty of these responses to know what I'm refering to, right? I know saying this will only encourage such responses but I feel better having said it.

Now...Shall we begin?


I've done everything I can think of and the server still eats 20-40% of the cpu when idling and 80%+ when only ONE player is connected. If you get more than one player on it will quickly bog down the whole server causing chat lag and "regenerating" blocks.

I've tried running the server with no plugins and a fresh map. Same deal. I've tried forcing the java VM to run in client-mode and server-mode doesn't make a difference. Tried purging anything to do with java and reinstalling; no luck. Tried using the 32bit java binary and ia32libs; no luck.

Now if I 'tar' up the entire minecraft server directory, download it to my home linux box and run it from there I get vastly different results. Idle cpu usage is only 0-3% and it only goes above 30% occasionally with 4 people connected. Here the only lag anyone gets is when a new player connects or teleports someplace causing my upload to be saturated with sending them the chunks.

The Minecraft console doesn't say anything helpful to me, anyways.
There are these occasional messages that appear in the console
[WARNING] Can't keep up! Did the system time change, or is the server overloaded?
[INFO] Attempted to place a tile entity where there was no entity tile!
This last one seems to show up rarely but always, at least, in pairs.
[INFO] New max size: <insert random number between 400 and 7000>

I did notice while gathering this info that entity id's are growing at an alarming rate, to me. As far as I know those go up linearly, correct? How is it in the first 2 minutes after a restart the 3 people that logged in get vastly differing entity ids. The first person (me and only connected to make sure you could actually connect) got entity id 86, the next person 15 seconds later got 4105. 1 minute, 45 seconds after that the 3rd person logs in and gets the entity id 75388. I'm thinking this is false positive though. My guess is NoLagg is stopping mob spawns successfully but the entity id still gets incremented. I'll have to test this later to make certain.

I've used htop to monitor the java process and threads. The java process and thread spends most of their time sleeping. Even when sleeping there is one thread that is eatting 20-40% of the cpu. Generally when I've seen a sleeping process/thread with high cpu usage it is because of an I/O bottleneck with the hdd. I don't know how to debug java at all to see where the problem is. I have no idea what thread it is beyond it is a sleeping thread and is eating a large amount to cpu time.


Now for the detailed information on what I'm running:

---------------------------------------------------------------------------------------------------

Minecraft version: 1.8.1
Bukkit version: 1240 1185
Plugins: noEnderman, dynmap, NoLagg, DefaultCommands, WorldEdit.
Start command: screen -dmS minecraft java -Xmx1024M -Xms1024M -jar craftbukkit-0.0.1-SNAPSHOT.jar nogui

---------------------------------------------------------------------------------------------------

Server System Info
OS: Linux 2.6.38-11-server x86_64 GNU/Linux

java version "1.6.0_26"
Java(TM) SE Runtime Environment (build 1.6.0_26-b03)
Java HotSpot(TM) 64-Bit Server VM (build 20.1-b02, mixed mode)

2GB RAM
Nic is connected at 1gbps but only alloted 50mbps of that (~6MB/s up and down).
hdd writes at 80MB/s and reads at 130MB/s
/proc/cpuinfo (Quad-core, hyper-threading off)

Show Spoiler Hide Spoiler

processor : 0
vendor_id : GenuineIntel
cpu family : 6
model : 44
model name : Intel(R) Xeon(R) CPU E5620 @ 2.40GHz
stepping : 2
cpu MHz : 2399.877
cache size : 12288 KB
physical id : 0
siblings : 1
core id : 0
cpu cores : 1
apicid : 0
initial apicid : 0
fpu : yes
fpu_exception : yes
cpuid level : 4
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat clflush mmx fxsr sse sse2 ht syscall nx lm constant_tsc rep_good nopl pni ssse3 cx16 sse4_1 sse4_2 popcnt hypervisor lahf_lm
bogomips : 4799.75
clflush size : 64
cache_alignment : 64
address sizes : 40 bits physical, 48 bits virtual
power management:

Minecraft takes 6-8 seconds to start-up during this time the server list will randomly show the correct information or "communication error" upon refresh. Connecting to the server will result in an immediate "Read Error: End of Stream" until the 6-8 seconds have passed then connections will be successful. Cpu usage is 160% to 240% (combined total of all cores in use by java)

On shutdown the server takes 2-3 seconds to kick players then another 8-12 shutdown completely and terminate. CPU usage doesn't change until it kicks the players then it jumps to ~200% (combined total of all cores in use by java)


---------------------------------------------------------------------------------------------------

Home Box System Info
OS: Linux 2.6.38-11-server x86_64 GNU/Linux

java version "1.6.0_26"
Java(TM) SE Runtime Environment (build 1.6.0_26-b03)
Java HotSpot(TM) Client VM (build 20.1-b02, mixed mode, sharing)

1GB RAM
Nic is connected at 1gbps but connection to my ISP is only 10mbps (~1.1MB/s down and 150KB/s up).
hdd writes at 20MB/s and reads at 47MB/s
/proc/cpuinfo (single-core)

Show Spoiler Hide Spoiler

processor : 0
vendor_id : AuthenticAMD
cpu family : 15
model : 44
model name : AMD Athlon(tm) 64 Processor 3000+
stepping : 2
cpu MHz : 2009.400
cache size : 512 KB
fdiv_bug : no
hlt_bug : no
f00f_bug : no
coma_bug : no
fpu : yes
fpu_exception : yes
cpuid level : 1
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 syscall nx mmxext fxsr_opt lm 3dnowext 3dnow up pni lahf_lm
bogomips : 4018.80
clflush size : 64
cache_alignment : 64
address sizes : 40 bits physical, 48 bits virtual
power management: ts fid vid ttp tm stc

Minecraft takes 1-2 seconds to start-up during this time the server will properly accepts connections but won't send them any chunk data at first. Cpu usage is 100%.

On shutdown it kicks everyone right then and there plus another 3-5 seconds to completely shutdown and terminate. Cpu usage bounces around.


---------------------------------------------------------------------------------------------------

I've looked about, managed to locate the problematic thread and it looks to be a vanilla Minecraft server issue?

I've gotten the stack in the middle of player joined world a number of times when no player had actually joined or left the world. When a creature entity enters the world it is a different stack trace.

Code:

"Server thread" prio=10 tid=0x00007f26fc07b800 nid=0x76f5 runnable [0x00007f2700939000]
   java.lang.Thread.State: RUNNABLE
        at org.bukkit.craftbukkit.util.LongHash.containsKey(LongHash.java:26)
        at net.minecraft.server.ChunkProviderServer.isChunkLoaded(ChunkProviderServer.java:45)
        at net.minecraft.server.World.isChunkLoaded(World.java:286)
        at net.minecraft.server.World.a(World.java:273)
        at net.minecraft.server.World.entityJoinedWorld(World.java:1183)
        at net.minecraft.server.WorldServer.entityJoinedWorld(WorldServer.java:48)
        at net.minecraft.server.World.playerJoinedWorld(World.java:1175)
        at net.minecraft.server.World.cleanUp(World.java:1090)
        at net.minecraft.server.MinecraftServer.h(MinecraftServer.java:460)
        at net.minecraft.server.MinecraftServer.run(MinecraftServer.java:374)
        at net.minecraft.server.ThreadServerApplication.run(SourceFile:417)

A creature entity trace.

Code:

"Server thread" prio=10 tid=0x00007f26fc07b800 nid=0x76f5 runnable [0x00007f2700939000]
   java.lang.Thread.State: RUNNABLE
        at net.minecraft.server.Entity.ao(Entity.java:686)
        at net.minecraft.server.EntityCreature.c_(EntityCreature.java:82)
        at net.minecraft.server.EntityLiving.s(EntityLiving.java:788)
        at net.minecraft.server.EntityLiving.s_(EntityLiving.java:280)
        at net.minecraft.server.World.entityJoinedWorld(World.java:1193)
        at net.minecraft.server.WorldServer.entityJoinedWorld(WorldServer.java:48)
        at net.minecraft.server.World.playerJoinedWorld(World.java:1175)
        at net.minecraft.server.World.cleanUp(World.java:1090)
        at net.minecraft.server.MinecraftServer.h(MinecraftServer.java:460)
        at net.minecraft.server.MinecraftServer.run(MinecraftServer.java:374)
        at net.minecraft.server.ThreadServerApplication.run(SourceFile:417)

I got this one once.

Code:

"Server thread" prio=10 tid=0x00007f26fc07b800 nid=0x76f5 runnable [0x00007f2700939000]
   java.lang.Thread.State: RUNNABLE
        at net.minecraft.server.EntityTracker.updatePlayers(EntityTracker.java:118)
        - locked <0x00000000c3ae0c50> (a net.minecraft.server.EntityTracker)
        at net.minecraft.server.MinecraftServer.h(MinecraftServer.java:469)
        at net.minecraft.server.MinecraftServer.run(MinecraftServer.java:374)
        at net.minecraft.server.ThreadServerApplication.run(SourceFile:417)

Most of the traces for the server thread where this here.

Code:

"Server thread" prio=10 tid=0x00007f26fc07b800 nid=0x76f5 sleeping[0x00007f2700939000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
        at java.lang.Thread.sleep(Native Method)
        at net.minecraft.server.MinecraftServer.run(MinecraftServer.java:351)
        at net.minecraft.server.ThreadServerApplication.run(SourceFile:417)

Other than the locked stack trace nothing really looks off. Well I guess the frequent playerjoined world traces are odd when there were no players entering or leaving.

---------------------------------------------------------------------------------------------------

For anyone that managed to survive all of that.

The only ideas I have left is a problem with the processor being a XEON or some obscure setting/package on the server box. Got any hints, suggestions, and/or fixes? I'd really appreciate it.


[edit]
Clarified that hyper-threading is off for the Xeon CPU.
Added start-up and shutdown times for both stations and how the server behaves during those times.

[edit2]
Found the responsible java thread.

[edit3]
Added a few more stack traces.

[edit4]
Updated to craftbukkit 1240. Idle CPU is still 20% but with three users connected CPU hovers around 50%. Which is an improvement. We'll see how it is doing after having ran for several hours.

[edit5]
Forgot to actually change the bukkit version to 1240 in the post after claiming I had in the previous edit. I'm sleepy, ok?

 

My server will be going fine then all of the sudden the CPU will spike to 99% and my server will give read timeout errors until I kill the java process and restart my server.
Is this like what your having? Or is your server not crashing? Just curious.

http://img651.imageshack.us/img651/6744/fsefsefk.png

 

The server initially upon start up gives everyone "read error: End of Stream" for maybe 6 seconds then it drops to the 20% cpu usage and start accepting connections properly.

The server start up so fast on my home linux box I'm not able to get a "read error: End of Stream"

 

Yeah, sometimes mine will start responding again and it will accept connections but about 60% of the time. Very annoying.

 

I've been getting really high CPU usage too. I can't track the cause either.

 

Which process is driving your cpu up guys, is it javaw.exe or java.exe (don't know if these are the same on that ghetto operating system called linux or not).

Also, is your CPU at high cycles when nobody is logged into the server?

 

....Let's leave the OS bashing elsewhere. Especially when you ask a question that was already answered.

javaw.exe and java.exe are the identical except for ONE thing. java.exe creates a console window for stdout and stderr output by the application running in java. javaw.exe does not create this console window for stdout and stderr output forcing the application to provide it's own method of conveying that information if it ever does.

For client applications you want javaw.exe unless you are debugging then java.exe is a bit more helpful.
For server applications you will probably always want java.exe unless the application creates it's own gui window as vanilla minecraft does unless you specify 'nogui'.

On linux there is only a need for binary named 'java' because whether or not there is a console depends on if it was launched from a console window or not. MacOS is likely the same as linux is this regard.

 

hmm, are those numbers for read and write actual numbers or max numbers? what are those numbers without running minecraft?

 

Wasn't sure if idle = 0 users on the server.

I've been rolling back and forward between 1.7 and 1.8 all day today, I'm pretty confident its a 1.8 issue rather than a bukkit issue. Were you having the same issues on 1.7?

Also, are you running the client off the "home box" ? I would say it needs more memory. As soon as I launch my client, it grabs 900mb. It frequently pushes around 1.3-1.5gb memory.

 

You talking about the hdd read/write speeds and the upload/download speeds? Those are without minecraft running. For both the server and client the hdd speeds are actual read/write speeds. For my home Internet connection those are the actual speeds. As for the server the download is correct but the upload speed is supposed to be 50mbps but I do not have a connection anywhere that can saturate it.

I thought the idle at 20% then one user is 80%+ was clear enough. My bad.

Yeah that is what I'm thinking after getting some stack traces. This server setup has been around since before 1.3 (1.3 is when I started playing mc.) No one has complained until 1.8. That said I don't know what the cpu usage was like before. with 1240 cpu usage hasn't changed when there are no players still ~20% but with players it is only about 40% now. Still unsure if the lag problems have gone away just yet.

Oh god no. My actual user desktop is an i7 with 6gb ram. Even if I was playing mc from that "home box" explain to us how it is still performing better than the much beefier server box? I knew AMD makes some efficient processors but can't be that good can they? I mean Intel took over what little market they lost to AMD with the core 2 duo.

 

I was talking about the hdd, so those numbers are the max your drives can do or those numbers are what your drives are doing atm.

If those numbers are what they are doing atm, are these with or without bukkit running?

 

I think I have finally managed to "solve" my problem a few days ago. It is an entirely stupid fix the way I see it. I don't know who's problem it is anymore. It only seems to affect bukkit. Vanilla Minecraft works fine and so does any other application on the server.

I managed to hook up VisualVM to the java process running bukkit. Immediately I'm seeing I/O wait times for sending packets to clients. Now I'm wondering if there is something I don't know about saturating the nic so I run iftop. Just like that the I/O wait times vanish and the cpu usage drops to <15%. This "fix" persists until the minecraft server is restarted. I don't know if the problem is bukkit, minecraft, java, the nic, or the nic's driver at this point. :/

I've setup the minecraft init script to launch iftop in another screen session a few seconds after starting the server then end the screen session a few seconds after that. Seems to work so far. I'll just still with this solution for now. I want to get back to actually playing the game for awhile. haha.

 

I am running my bukkit server in server 2008 and I am having the same problem, within 20 mins of the server up and only a total of 4 players on the last player to log on had the entity id 40251.

Then the server keeps saying cant keep up error message then the cpu usage jumps to 25% of a quadcore 3.2GHz xeon and then it eats up another gig of ram and kicks everyone.

I have no idea how to fix this.

Any help would be great, thanks.

 

my server loads and works but my minecraft freezes when i join it