Bukkit Persistance reimplemented - no code changes required!

Discussion in 'Plugin Development' started by LennardF1989, Jul 6, 2011.

Thread Status:
Not open for further replies.
  1. Offline

    xpansive

  2. Offline

    LennardF1989

    Give me a moment to check it out.

    @xpansive Your stacktrace says something about calling VoronoiNoise.Save() in OnDisable, but the code on GitHub doesn't have any of that. Can you upload the latest version which gave you that error on /reload (which I guess you attempted to do).

    But I also see what you are doing wrong. You're trying to persist the Data-class, with Lists, but didn't specify a Foreign Key, besides that, Point and Point3D aren't Ebean Entities, it will not be able to save those.

    I'll make a rough version on your code, have a moment.

    EDIT by Moderator: merged posts, please use the edit button instead of double posting.
     
    Last edited by a moderator: May 17, 2016
  3. Offline

    xpansive

    About the onDisable thing, I changed it to onWorldSave in ExpansiveTerrainEventListener.java. You need to do /stop now to get the stacktrace, here's the updated one: http://pastebin.com/JZFdrYcF
    Sorry for any confusion that may have caused.

    Anyways, how would I make them Ebean Entities? I thought they just needed to be serializable.
     
  4. Offline

    LennardF1989

    @xpansive Ok, I updated your project and attached it to this post. Have a look, I commented why I did what :)

    As for your plugin, it has a few flaws:
    1. It will generate database/config files OUTSIDE the plugins directory, you're going to annoy a lot of people if your final plugin does this, too. I know it throws a few errors at you for doing it correctly, but with some debugging you should be able to fix it ;)
    2. Don't use static when you don't have to.
    3. Your plugin is unreliable. In case your server crashes, data will never be saved. What you should do is:
      1. OnEnable load the database into memory and retrieve ALL Data-objects from it, store them in a local variable.
      2. When a new Chunk is added to a Data-object, do:
      Code:java
      1. Point3D point3d = new Point3D(0, 0, 0);
      2. dataList.get(0).getChunks().add(point3d); //0 = World 1, for example
      3. getDatabase().persist(point3d); //Do the same for the 2D points
      This way, your data will be instantly saved and you will never lose anything. It's also a cheap way of storing data, as you only save the new point, not the full array.
    Also, a screenshot of your database-structure as it should be with my code:

    Tables.png
     

    Attached Files:

  5. Offline

    Acrobot

    @LennardF1989
    Hey, sorry to bother you :p
    http://pastebin.com/0ABLWBxw
    Line 120 of ChestShop.java
    Code:
            database.initializeDatabase(
                    getFromConfiguration("database.driver"),
                    getFromConfiguration("database.url"),
                    getFromConfiguration("database.username"),
                    getFromConfiguration("database.password"),
                    getFromConfiguration("database.isolation"),
                    false,
                    false
            );
    
     
  6. Offline

    LennardF1989

    The error comes from the place where it prepares the URL, debug if all the getFromConfiguration-statements actually return something. My guess is that "database.url" returns null.

    EDIT: Also, not that I really care, but, do you have problems with the "My" in the class file :p?
     
  7. Offline

    Acrobot

    @LennardF1989
    Oh well, I'm sorry.
    I have accidentally read incorrect config file -.-
     
  8. Offline

    LennardF1989

    @Acrobot
    I looked into your issue with UTF-8 characters. I tried, but there is no way around the issue. Supposedly your OS determines the charset/locale Java uses. Although, this doesn't seem to work properly (mine is Dutch, so it SHOULD give me access to éáà-character etcetera).

    The best remedy is using the command-line argument in absence of a better solution. A nice topic on the issue can be found here, in case your interested: http://stackoverflow.com/questions/1749064/how-to-find-default-charset-encoding-in-java
     
  9. Offline

    Acrobot

  10. Offline

    xpansive

    Wow, thanks for doing all that... I'll have a look at it right away.

    Edit: The reason the database and config files are stored outside of the plugins directory is because I wanted them stored with the generated world so you can just delete the world folder to get a new one.
     
  11. Offline

    LennardF1989

  12. Offline

    Acrobot

    @LennardF1989
    Not really, but I have removed it (just for a moment), because I currently have problems with my configuration class (almost resolved)
     
  13. Offline

    triggerhapp

  14. Offline

    phaed

    When my plugins used Bukkit's built in Ebeans persistence I ran the jvisualvm profiler on the minecraft server and found that Ebeans alone sitting idle took 60% of the processing time of the server. I assume this hasn't changed with this setup?
     
  15. Offline

    LennardF1989

    @phaed I haven't experienced that sort of behavior in my plugins, could you describe in what sort of fashion the persistence layer was used? Like, how many Ebeans, how many foreign keys, how many lists, etc? Was it one of your plugins or someone else's?

    Were you using scheduler for a repeating task with small delay? Because I noticed a huge processing power drain when doing that. Threads?

    Sorry for the question bomb but it is in my believe everything can be fixed... somehow.
     
  16. Offline

    phaed

    No scheduler that I can think of, it was a 5 table setup with one of the tables referencing another which referenced another. You can decompile a version 4.x of PreciousStones and take a look http://sacredlabyrinth.net/releases/
     
  17. Offline

    croxis

    @LennardF1989 Do you mine providing an example on how updating a database would look like. I assume I would inherit MyDatabase into my own class and override afterCreateDatabase?
     
  18. Offline

    LennardF1989

    @croxis I'll put up an example somewhere first thing tomorrow!

    Sorry for the late reply.
     
  19. Offline

    croxis

    @LennardF1989 I finally had a chance to look closer at the code, and beforeDropDatabase and afterCreateDatabase are only called if the db is being rebuilt. So if my plugin detects that it needs to modify the database:

    1) Load database into memory as normal and handle the exceptions from the new schema files in a sane manner.
    2) Unload the database.
    3) Reload the database with rebuild set to true
    4) Save the contents of memory back into the database
     
  20. Offline

    LennardF1989

    Exactly.

    I recommend the following:
    - Store the version of your plugin in a local config file, so it's the first thing read into memory when the plugin starts.
    - If "version in config" < "version of plugin" => expect upgrade.

    Programmically make a backup of your database first. Rebuild is DESTRUCTIVE!

    Now, you have two choices as for upgrading:

    #1
    - Create extension of MyDatabase SPECIFICALLY for that version and register the old unmodified Ebean-classes (yes, you have to keep those in your project alongside the new versions).
    - Create another extension of MyDatabase for your new versions of the Ebeans.

    Obviously the old database loads fine, no events are fired. The second, however, doesn't exist yet and will fire the onDropDatabase/afterCreateDatabase events. Because you are expecting an upgrade and have the old database in memory as well, you can reinsert data in the new format in the afterCreateDatabas-event.

    #2
    - Have both the old Ebean as new Ebean versions in your plugin, but instead of making 2 database-instances, give your new version a different Table-name (say you have a table world of your old version, create a second Ebean with world_v2 as a name).
    - Register both the old and new classes in the same MyDatabase extension.
    - Set rebuild to true, this will cause events to fire.
    - When onDropDatabase gets called, retrieve all old data-records (eg. World.getAll()) and convert them to the new format and store it in memory (eg. new World_v2(world.name, world.x, world.y, world.z,)).
    - Now the database is dropped and cleaned
    - When afterCreateDatabase is called, reinsert the new data (eg. database.Save(listWordsv2)).

    And you're ready to go.

    Better yet is providing an "upgrade plugin", which does just that, so you can keep the actual plugin clean.
     
  21. Offline

    croxis

    Ugg, my problem is I'm adding support to Towny, already set in its non-relational database ways. I'm kind of baffled that it is so difficult to go "new_field doesn't exist? ALTER TABLE my_table ADD new_field VARCHAR(60)!" automatically.
     
  22. Offline

    LennardF1989

    It's the trade-off we pay over much more CPU intensive Persistance Layers like Hibernate. Besides that, SQLite's limited ALTER TABLE statement also doesn't help much.
     
  23. Offline

    croxis

    SQLalchemy does it :p
     
  24. Offline

    Rycochet

    Using this in the mmoMinecraft suite now - but due to shared config options can't use rebuild=true, so made a slight change to automatically create the database the first time it's run -

    https://github.com/LennardF1989/BukkitEx/pull/1

    Basically moved the check in installDatabase() to after seeing if the tables exist, and if they don't it'll create them. If there's only one table missing, then it won't try to create that single table - only if none of them are there.
     
  25. Offline

    Acrobot

  26. Offline

    Rycochet

    I could see using it to update to a new version - simply rename and then open a v1 database, then open a v2 database - copy everything over, close v1, then delete... Just got to automate it ;-)
     
  27. Offline

    LennardF1989

  28. Offline

    Ycros

    This is fantastic, why don't you try to get this pushed upstream into Bukkit itself?
     
  29. Offline

    LennardF1989

    Because they make no haste in accepting pull requests. I still have two requests open from about 2 months ago and while the contents are discussed with some community members, none of the actual Bukkit devs replied to any of the request - heck, even a 'We don't need it, but thanks anyway' would have been satisfying for me.

    I therefore make no haste myself to get this into Bukkit. I also have much more control over bug-fixing when the class is not sealed into Bukkit.

    EDIT: To not be that much of a hypocrite, I want to say sorry to the open pull request on MyDatabase, I'll put it on my schedule for tomorrow (I am a busy man :p).
     
  30. Many thanks LennardF1989 for putting this code up. I'm using it to wrap the database access in my plugin too.

    Has anyone figured out a good way to add extra columns to existing tables when a new column is needed (forcing users to delete the .db and start again isn't really an option).
    This was easy enough to detect and do when I was doing all the database management in MySQL but I've not figured it out yet with all the wrapping around the code (MyDatabase/EbeanServer/sqlite/JPA etc).
     
Thread Status:
Not open for further replies.

Share This Page