Random Lag Spikes That Kick Everyone

Usually I can fix most the problems I encounter on my own or with a quick Google, but I'm at a loss for this one.

For about 24h, my server has been randomly freezing. There are no errors or any tell-tale signs that it is about to happen or has happened, besides every player suddenly getting and end of stream error.

I'll show you what I mean, here are the logs from of the freezes:

Everything is going normal, then it freezes around 01:18:45, and 20 seconds later endOfStreams start spamming the console as every player loses their connection. This has probably happened ~15 times today.

At first, I thought it could be a memory leak or something else forcing the server to do intensive garbage collection to free RAM, but RAM usage is completely normal before and after the freeze.

Then I thought that it could just be the CPU getting hammered by tons of tasks at once, so I checked the 1, 5, and 15 minute CPU usage figures right after a crash. There was no significant difference between the two, so I don't think it's the CPU.

Now I think I've got it down to two possibilities: the network or a plugin sleeping the main thread. I only have rough graphs, so I can't speak with complete accuracy, but I have not noticed any sudden spikes in incoming or outgoing traffic. As for plugins, I have not made any significant changes recently, so I don't see why it would have suddenly started in the past 24h.


TL;DR: I keep having a mysterious freezing and I can't locate the source. Does anybody have any similar experiences? How did you end up solving it?


EDIT: I made a new observation. Whenever this freeze happens, I am unable to ssh into the server or ping it until the freeze ends.

 

Having the exact same since 1.4.2, doesn't matter which beta or dev build we used at the time, currently #2455.

Box runs completely fine, player count does not matter, but every few hours or so, completely out of the blue, half the players get the error exactly as in your log. RAM and CPU look very good, nowhere near being maxed out, it's a dedicated CentOS in a data centre, and I think I know my stuff quite well, too.

The only significant change was the switch to 1.4.2. It appears to happen when certain players log in. By certain players I don't mean a corrupt player.dat or that their location is loaded with entities or something, and it's different players all the time, but the other players said they get d/c when "player xyz logged in".

I wouldn't call it a freeze though, console works completely normal when this happens. A quick server restart always solves the issue immediately and completely. So I'm not thinking it's the network being hammered or that the data centre suddenly has random spikes or whatnot. It happens every few hours, maybe 4, maybe 8, it's kind of random.

My best guess so far is 1.4.2, or a plugin that doesn't work too well with it, yet. Which one? Not the faintest clue.

tl;dr: I think I have the same problem.

 

I have been having the same issues as well.

 

Happened twice to me as well, only since 1.4.2. :s

 

Turn online mode to true. We can't help a cracked server.

 

My server is in online mode. Please use common sense if you're going to post.

I updated to the latest dev build, and I think it fixed it. Or at least it greatly reduced the frequency.

EDIT by Moderator: merged posts, please use the edit button instead of double posting.

 

The server I'm plugin developer on has this happening a lot lately. It usually gets better a couple of minutes after a restart.

 

Which build did you use before?

 

Latest beta build: http://dl.bukkit.org/downloads/craftbukkit/view/01542_1.4.2-R0.2/

 

I have the same problem for few days, its very annoying I'll try latest dev build how thebeast said

 

But according to the changelog, nothing changed since that beta build (#2455) and the dev you are using, #2457. They merely changed the version number, no actual code changes or bug fixes are mentioned. (As they always do after a RB or Beta)

Is it technically possible that this could in any way be related to Mojang's servers? I don't mean the brutal down times, but would that technically be possible, that players get disconnected even if the login process itself happened minutes or half an hour earlier?

 

No, that is not possible. It authenticates once, then leaves the mojang server for what it is.

 

Yes, thought so. But this thing is so random, got to go wild in the search for the reason.

Btw, problem has not happened for over 12 hours now. Why? Absolutely clueless, didn't change a thing.