Util Persistent Data Past Reload (v0.1)

So today I bring you something special. It can be extremely powerful, or extremely harmful, depending on how it is used and the user behind it. I don't know why I am releasing this tool, but it may be because I'm tired of plugins that cause my server to freeze for 10 full seconds during a reload. Or maybe it's because I'm a jerk and I like being first to publicly create stuff. Anyways, this tool, dubbed, for a lack of a better name, UnsecureClassloader, allows for some special stuff to happen. It allows for classes to be loaded preplugin in the ClassLoader Hierarchy. The significance of this, is that it allows devs to reconnect to threads created and recover data during reloads without major reflection. The simplicity of this tool allows even a beginner to use it (correctly I hope).

UnsecureClassloader.java

Example Uses:
ReloadCounter.jar (Source)

How to Use:

Create a custom (Data)Thread. This thread is used to store data, and then reconnect to the thread and retrieve the data. This class created is special in the fact that it needs to be loaded with our utility or else we wont be able to reconnect to the thread and our data will be forever lost (unless you take the long way and use reflection to get the fields and such). Make sure it has a run method which makes it sleep, and a end method which interrupts its sleeping, and a way to store information. Be aware of what you store in here, because anything that is loaded with your plugin (ex: other classes), wont be accessible because of casting exceptions in the Class Loader. If you don't understand what I mean by anything loaded with your plugin: look at your project in Eclipse. See the files that you created? Anything that isn't preloaded with the Utility, will be inaccessible when you try reattach to the thread.

DataThread.java (open) DataThread.java (close)

Code:

package com.deadmen.example.dataplugin.resources;

public class DataThread extends Thread{

    public DataThread(String name){
        super(name);
    }
  
    public DataThread(String name, DataObject dO){
        super(name);
        this.dO = dO;
    }
  
    private DataObject dO;
  
    public DataObject getDataObject(){return dO;}
  
    @Override public void run(){
        while(!this.isInterrupted()){
            try{
                Thread.sleep(10000000);
            }catch(Exception e){break;}
        }
    }
  
    public void end(){
        this.interrupt();
    }
  
    public static DataThread retrieveDataThread(String name){
        try{
            for(Thread t: Thread.getAllStackTraces().keySet()){
                if(t.getName().equals(name)) return (DataThread)t;
            }
        }catch(Exception e){e.printStackTrace();}
        return null;
    }
  
}

Next we want to create a "DataObject" class. The point of this class is to allow an easy mount to retrieve data. You can add any field that you would like, that is already loaded (ex: what you preload, and bukkit classes). NOTE TO NOT USE THIS CLASS FOR METHODS. IT SHOULD REALLY ONLY BE USED FOR STORING FIELDS. IF YOU USE METHODS IN THIS CLASS, MAKE SURE TO HAVE AT LEAST A BASIC UNDERSTANDING OF CLASSLOADER HIERARCHY!

DataObject.java (open) DataObject.java (close)

Code:

package com.deadmen.example.dataplugin.resources;

public class DataObject {

    //...
}

Finally we come to our Main method, which most of the programmers here seem to name their plugin's name (I name it Main so its easy to find, and because it houses the main starting point of the program). onLoad, we want to check if the classes are loaded. To do this, we use the tool's checker methods, which return a boolean. If they aren't loaded, then load them using the proper methods and init the DataObject field. Make sure not to init the DataObject field before the class is loaded with the Util, or else it will cause you to crash. If the classes are loaded, then try to reconnect to the thread which stores the information. If you connect to it, then get the information and store it in the DataObject field. Then end the thread. You want the thread to die so its taking up less memory and processing power than needed. We only needed it to be persistent past a reload, so it's done it's job. onDisable, we want to wrap the DataObject in a new Thread, and start it. This will restart the Cycle. Take note, that you can't reuse threads, and once they are terminated, they are dead for good.

Main.java (open) Main.java (close)

Code:

package com.deadmen.bukkit.main;

import org.bukkit.plugin.java.JavaPlugin;

public class Main extends JavaPlugin {

    private static final String threadName = "DataThreadForPlugin";
  
    private static DataObject DO;
    public static DataObject getDataObject(){return DO;}
    public static void setDataObject(DataObject dO){DO = dO;}
  
    public void onDisable() {
        new DataThread(threadName,getDataObject()).start();;
    }
    public void onLoad() {
        if(!UnsecureClassloader.classDefined("com.deadmen.example.dataplugin.resources.DataObject")) { //check to see if one of, or more than one, of the classes were loaded in.
            try{
                UnsecureClassloader.loadPackage(this.getFile(), "com.deadmen.example.dataplugin.resources"); //Load the package that houses the resources that are needed
                DO = new DataObject();
            }catch(Exception e) {e.printStackTrace();}
        }else {
            DataThread d;
            if((d = DataThread.retrieveDataThread(threadName))!= null) {
                DO = d.getDataObject();
                d.end();
            }else {
                DO = new DataObject();
            }
        }
    }
}

That's all there really is to it. If anyone wants to improve on anything here go right ahead. The Utility, created by CyanFrost and I, is free for anyone to use, but I would love to see one plugin be used as a master persistence store so we don't have a large amount of threads when they aren't really needed.

 

@MrAwellstein O boy, look at that memory leak you posted.

 

@MrAwellstein
https://github.com/ddopson/java-class-enumerator - needs a few tweak

 

Please only have constructive criticism, instead of just blind, hateful comments, if you will. If you want to say something bad about my Util, then please support your claim with reason and fact.

 

Cleaned up thread

 

Thank you.

 

It's not a blind, hateful comment. You leak memory. Your whole concept is about leaking memory. And the fact that your plugin works proves that:

• Bukkit is supposed to unload your classes and load them from disk.
• This also means your class is loaded from disk, and is assigned to another part of memory than your first load.
• In turn, assigning the ReloadCounter from the Thread to your field should cause a ClassCastException
• But it doesn't. This means that your initial ClassLoader and the new ClassLoader (reloads create new ClassLoaders) are both in memory => nice memory leak you got there.

 

My ClassLoader isn't a real ClassLoader. My classloader, is actually unloaded with the server on each reload. The purpose of it, is to put one or two classes into the Main JVM classloader, which, isn't a memory leak because they aren't constantly readded and they get removed when the JVM turns off. The purpose of this is to prevent the ClassCastException, by making the Class declared before the plugin is, making it like a dependency. Your statement makes me think that you posted this without actually looking at my code :/

Edit: I wasn't the one who named it a "ClassLoader", but I was under the impression that it was named UnsafeClassloader because it was a virtual ClassLoader which used reflection to use classes.

 

I wasn't talking about yours, I was talking about Bukkit's .

And they will stay there even if you remove the plugin => memory leak. Plugins are not supposed to have memory claimed after they are removed, it's called bad memory management

While the trick itself is pretty smart, the fact that you can't remove the plugin is pretty bad and should be enough reason not to resort to your trick. If you so desire to be able to reload a plugin without that 10 second delay, you could also start by writing it in a way that allows you to load data without freezing the main thread (decent asynchronous programming, perhaps?)

Such a shame that people think I would criticize without reading source code...

 

You forgot to mention that #onLoad() is actually called after plugin instantiation, which makes the claim of declaring the class before the plugin invalid. static blocks will execute before #onLoad() will

if you want true persistence, be a real programmer (yes i totally jacked this from a convo with cyan and another user), and use bytecode injection (eg. instrumentation) to attach a persistence framework to bukkit as to allow plugins to have persistent data, without all the overhead and hacky solutions (one of which you provided)

 

I never actually said that this would actually load before the plugin. But in technicality it does load before the plugin, after the first reload. Also bytecode injection isn't needed here. Oh, and FYI, I am a real programmer.

It's supposed to stay there. Its intentional, so its not technically a leak. Also you can remove the plugin, just not the classes that are injected, which is also intentional. Also your remark about Asynchronous Programming, the main purpose of this was to keep Sockets alive and running, which, don't like to be abruptly ended and restarted without interrupting the user. And yes I do feel like you criticized without reading the source.

I meant in the classloader hierarchy, not in the actual time.

 

I totally respect @fireblast709 (I don't really know @Not2EXceL), but this utility is badass. I'd like to do some more in-depth testing with it, but just having something like this publicly available (to those who would know how to properly utilize it), is very nice.

 

Well, wikipedia disgreed with you on that one.

Well implement a reconnect function on the client side...

Clean up your garbage. If you don't want to, well, don't create garbage in the first place.
(note, because I know you are going to take this offensive, garbage refers to the memory leak, not the resource)

 

I never said you said that it loaded before the plugin. You stated that this declared your class before the plugin's declared, which according to classloader heirarchy the class extending Plugin, will be loaded, then static blocks ran, and finally the instantiation, which then your #onLoad() function runs after all of that happens.

*the real programmer was taken from a convo between your partner on this util and adam, seriously you too that way too seriously. It wasnt meant to be direct or attacking in any way.

The classes aren't injected at all. Use proper terminology. You're just using a hackish solution to load the class into Bukkit's classloader instead of the classloader bukkit creates for each plugin.

And regarding your statement on bytecode injection isn't needed. Your right, it isn't, but then again neither is a hackish solution. If anything one should use what will in the end be most stable and instrumenting a codecave into bukkit will not only be safer memory wise, but also easier extend a framework for others to use, without the reliance upon a plugin loaded by bukkit.

On the side note, make the UnsafeClassloader (which really should be named to something like BukkitClassLoaderUtil or something) only able to load classes that implement a specific empty interface. I shouldn't have to explain why (as its extremely simple) or how you leave the option of loading any class into the bukkit classloader can cause problems.

(note, because I know you are going to take this offensive, everything hearby written not only contains opionions and not implications, the inferences made from said post is completely up to the reader and therefore the author is not at fault of anything) *idea totally not stolen from mark

 

I know that you meant the memory, when you meant garbage.
ALSO: that quote said "when isn't needed", which, it is always needed.
Also even reconnect functions disrupt the user.

I'll try to convince my partner to change the name.
I was thinking that too, but then I realized something: If someone uses my tool, they can simply remove any safety features I add: IE: preventing org.bukkit or net.minecraft packaged classes in.

I feel as if this hackish solution (which technically isn't all that hacky) safer than bytecode injection. I'll try to change to correct terminology

You really don't understand how much I agree with you on this one.


Furthermore, I want to know a better way to do this, if you have one, because I don't think one exist without modifying bukkit. But correct me if I'm wrong ^_^

EDIT by Moderator: merged posts, please use the edit button instead of double posting.

 

My main point being if I remove the plugin, how is it still needed . That's where your memory management fails - it will never be GCed (yes, after the JVM has shut down. But that's not what should happen, you should not keep that memory allocated when it's not used)

 

The only thing i gotta say here, is y not just PR a framework to bukkit?

 

Lol. Who do you think is around, still accepting PRs?

 

From my understanding, you cant remove the jar until the JVM has shutdown and is no longer using the Jar file itself, because Bukkit is still technically using it (unless you do it during a reload which I think you have low chances of doing it). Anyways, there is a cost to use certain plugins, and server admins must understand how to use them and how to properly manage them (shut down the server before removing a plugin).

 

It was more of a thing about how if you're gonna make a plugin, but it aint gonna be a plugin, and should just be moved to the core, then wat. makes less sense not to, and also didnt 1.8.1 bukkit come out after everything hit the fan?

 

No.

 

Well that's a flaw in Bukkit then, no need to abuse that in your argumentation to use hacky tools . I've written a plugin that allows me to dynamically load modules before (and no, the source was never published) that allows me to unload them dynamically, so this issue you are referring to is strictly Bukkit's issue.

See my previous statement. A decent version of this, also the challenge of your fellow developer CyanFrost (make sure you inform him about this), would be using the Bukkit API (modified or not).

Here ya go, all persistent within the scope of a reload, let us all sing the Metadata song. O yes, Metadata all the way.

 

unsafe.defineclass(rootloader, yourclass) that all you need - single line.

 

I'll do testing with that and update my code. But my Util allows an easy way to get the bytes and actually load them from a jar, rather than having suspicious byte[] somewhere in your code.

 

native layout of all data types in java is byte[], you can fool yourself by making multiple layers of wrappers, but this wont change how jvm works.