Bukkit Persistance reimplemented - no code changes required!

It doesn't need to be PM'd as its a open source project
https://github.com/xpansive/ExpansiveTerrain/tree/persistance

 

Give me a moment to check it out.

@xpansive Your stacktrace says something about calling VoronoiNoise.Save() in OnDisable, but the code on GitHub doesn't have any of that. Can you upload the latest version which gave you that error on /reload (which I guess you attempted to do).

But I also see what you are doing wrong. You're trying to persist the Data-class, with Lists, but didn't specify a Foreign Key, besides that, Point and Point3D aren't Ebean Entities, it will not be able to save those.

I'll make a rough version on your code, have a moment.

EDIT by Moderator: merged posts, please use the edit button instead of double posting.

 

About the onDisable thing, I changed it to onWorldSave in ExpansiveTerrainEventListener.java. You need to do /stop now to get the stacktrace, here's the updated one: http://pastebin.com/JZFdrYcF
Sorry for any confusion that may have caused.

Anyways, how would I make them Ebean Entities? I thought they just needed to be serializable.

 

@xpansive Ok, I updated your project and attached it to this post. Have a look, I commented why I did what

As for your plugin, it has a few flaws:

1. It will generate database/config files OUTSIDE the plugins directory, you're going to annoy a lot of people if your final plugin does this, too. I know it throws a few errors at you for doing it correctly, but with some debugging you should be able to fix it
2. Don't use static when you don't have to.
3. Your plugin is unreliable. In case your server crashes, data will never be saved. What you should do is:
   1. OnEnable load the database into memory and retrieve ALL Data-objects from it, store them in a local variable.
   2. When a new Chunk is added to a Data-object, do:
   Code:java

   Point3D point3d = new Point3D(0, 0, 0);
        dataList.get(0).getChunks().add(point3d); //0 = World 1, for example
         getDatabase().persist(point3d); //Do the same for the 2D points

   This way, your data will be instantly saved and you will never lose anything. It's also a cheap way of storing data, as you only save the new point, not the full array.

Also, a screenshot of your database-structure as it should be with my code:

 

@LennardF1989
Hey, sorry to bother you
http://pastebin.com/0ABLWBxw
Line 120 of ChestShop.java

Code:

        database.initializeDatabase(
                getFromConfiguration("database.driver"),
                getFromConfiguration("database.url"),
                getFromConfiguration("database.username"),
                getFromConfiguration("database.password"),
                getFromConfiguration("database.isolation"),
                false,
                false
        );

 

The error comes from the place where it prepares the URL, debug if all the getFromConfiguration-statements actually return something. My guess is that "database.url" returns null.

EDIT: Also, not that I really care, but, do you have problems with the "My" in the class file ?

 

@LennardF1989
Oh well, I'm sorry.
I have accidentally read incorrect config file -.-

 

@Acrobot
I looked into your issue with UTF-8 characters. I tried, but there is no way around the issue. Supposedly your OS determines the charset/locale Java uses. Although, this doesn't seem to work properly (mine is Dutch, so it SHOULD give me access to éáà-character etcetera).

The best remedy is using the command-line argument in absence of a better solution. A nice topic on the issue can be found here, in case your interested: http://stackoverflow.com/questions/1749064/how-to-find-default-charset-encoding-in-java

 

@LennardF1989
Thanks

 

Wow, thanks for doing all that... I'll have a look at it right away.

Edit: The reason the database and config files are stored outside of the plugins directory is because I wanted them stored with the generated world so you can just delete the world folder to get a new one.

 

@Acrobot @xpansive @triggerhapp
Encountered any problems or alike I need to take a look at?

 

@LennardF1989
Not really, but I have removed it (just for a moment), because I currently have problems with my configuration class (almost resolved)

 

@LennardF1989

Not a single one still. Good stuff

 

When my plugins used Bukkit's built in Ebeans persistence I ran the jvisualvm profiler on the minecraft server and found that Ebeans alone sitting idle took 60% of the processing time of the server. I assume this hasn't changed with this setup?

 

@phaed I haven't experienced that sort of behavior in my plugins, could you describe in what sort of fashion the persistence layer was used? Like, how many Ebeans, how many foreign keys, how many lists, etc? Was it one of your plugins or someone else's?

Were you using scheduler for a repeating task with small delay? Because I noticed a huge processing power drain when doing that. Threads?

Sorry for the question bomb but it is in my believe everything can be fixed... somehow.

 

No scheduler that I can think of, it was a 5 table setup with one of the tables referencing another which referenced another. You can decompile a version 4.x of PreciousStones and take a look http://sacredlabyrinth.net/releases/

 

@LennardF1989 Do you mine providing an example on how updating a database would look like. I assume I would inherit MyDatabase into my own class and override afterCreateDatabase?

 

@croxis I'll put up an example somewhere first thing tomorrow!

Sorry for the late reply.

 

@LennardF1989 I finally had a chance to look closer at the code, and beforeDropDatabase and afterCreateDatabase are only called if the db is being rebuilt. So if my plugin detects that it needs to modify the database:

1) Load database into memory as normal and handle the exceptions from the new schema files in a sane manner.
2) Unload the database.
3) Reload the database with rebuild set to true
4) Save the contents of memory back into the database

 

Exactly.

I recommend the following:
- Store the version of your plugin in a local config file, so it's the first thing read into memory when the plugin starts.
- If "version in config" < "version of plugin" => expect upgrade.

Programmically make a backup of your database first. Rebuild is DESTRUCTIVE!

Now, you have two choices as for upgrading:

#1
- Create extension of MyDatabase SPECIFICALLY for that version and register the old unmodified Ebean-classes (yes, you have to keep those in your project alongside the new versions).
- Create another extension of MyDatabase for your new versions of the Ebeans.

Obviously the old database loads fine, no events are fired. The second, however, doesn't exist yet and will fire the onDropDatabase/afterCreateDatabase events. Because you are expecting an upgrade and have the old database in memory as well, you can reinsert data in the new format in the afterCreateDatabas-event.

#2
- Have both the old Ebean as new Ebean versions in your plugin, but instead of making 2 database-instances, give your new version a different Table-name (say you have a table world of your old version, create a second Ebean with world_v2 as a name).
- Register both the old and new classes in the same MyDatabase extension.
- Set rebuild to true, this will cause events to fire.
- When onDropDatabase gets called, retrieve all old data-records (eg. World.getAll()) and convert them to the new format and store it in memory (eg. new World_v2(world.name, world.x, world.y, world.z,)).
- Now the database is dropped and cleaned
- When afterCreateDatabase is called, reinsert the new data (eg. database.Save(listWordsv2)).

And you're ready to go.

Better yet is providing an "upgrade plugin", which does just that, so you can keep the actual plugin clean.

 

Ugg, my problem is I'm adding support to Towny, already set in its non-relational database ways. I'm kind of baffled that it is so difficult to go "new_field doesn't exist? ALTER TABLE my_table ADD new_field VARCHAR(60)!" automatically.

 

It's the trade-off we pay over much more CPU intensive Persistance Layers like Hibernate. Besides that, SQLite's limited ALTER TABLE statement also doesn't help much.

 

SQLalchemy does it

 

Using this in the mmoMinecraft suite now - but due to shared config options can't use rebuild=true, so made a slight change to automatically create the database the first time it's run -

https://github.com/LennardF1989/BukkitEx/pull/1

Basically moved the check in installDatabase() to after seeing if the tables exist, and if they don't it'll create them. If there's only one table missing, then it won't try to create that single table - only if none of them are there.

 

@Rycochet
I did the same, just without moving anything
if(databaseExists) return;
:​D
https://github.com/Acrobot/ChestShop-3/blob/master/com/lennardf1989/bukkitex/Database.java#L188

 

I could see using it to update to a new version - simply rename and then open a v1 database, then open a v2 database - copy everything over, close v1, then delete... Just got to automate it ;-)

 

That's exactly how I intended the "update databases" part

I wrote a more extensive post regarding it on the previous page (please don't mind the typos, I guess it was early in the morning when I typed that): http://forums.bukkit.org/threads/bu...ode-changes-required.24987/page-2#post-579786

EDIT:
Fixed (most of) the typos in the post, lol.

 

This is fantastic, why don't you try to get this pushed upstream into Bukkit itself?

 

Because they make no haste in accepting pull requests. I still have two requests open from about 2 months ago and while the contents are discussed with some community members, none of the actual Bukkit devs replied to any of the request - heck, even a 'We don't need it, but thanks anyway' would have been satisfying for me.

I therefore make no haste myself to get this into Bukkit. I also have much more control over bug-fixing when the class is not sealed into Bukkit.

EDIT: To not be that much of a hypocrite, I want to say sorry to the open pull request on MyDatabase, I'll put it on my schedule for tomorrow (I am a busy man ).

 

Many thanks LennardF1989 for putting this code up. I'm using it to wrap the database access in my plugin too.

Has anyone figured out a good way to add extra columns to existing tables when a new column is needed (forcing users to delete the .db and start again isn't really an option).
This was easy enough to detect and do when I was doing all the database management in MySQL but I've not figured it out yet with all the wrapping around the code (MyDatabase/EbeanServer/sqlite/JPA etc).