"Supposedly there is a monster glitch where monsters from unloaded chunks aren't getting unloaded as well... causing buildup of unnecessary monsters over time, thereby eating more memory, until server crashes." -fullwall
Hmm.. only thing i can think of would be to occasionally use worldedit's Butcher command.. Something like /butcher 1500 Might help a bit, but definitely not a great fix.
The plugin developers are waiting for creature_spawn event to be implemented to Bukkit, so they can create a mob control plugin. I'm currently waiting for this one: http://forums.bukkit.org/threads/indev-crowdcontrol-0-10.4073/