[INACTIVE][DEV] Persistence v0.76 - Data-Driven Bukkit [677]

Discussion in 'Inactive/Unsupported Plugins' started by NathanWolf, Jan 29, 2011.

  1. Offline

    NathanWolf

    Persistence - Data-Driven Bukkit

    PLEASE NOTE
    Persistence is a developer API and framework- NOT a game or admin plugin.

    If you've come here looking for plugin/admin support for Spells, Wand, NetherGate or any other plugin that uses Persistence- please turn around and go back to that thread. Thank you for your cooperation!

    If you're a dev, then check the Persistence wiki for information on using the Persistence framework.

    View changelog on github
     
    Duccis likes this.
  2. Offline

    NathanWolf

    Send me the code when you've got that- I've always wanted to make a persisted version of location.

    If you stick a BlockVector, an Orientation (built-in Persistence thing) and a WorldData in there, call it LocationData- I'll put it in Persistence core :D
     
  3. Offline

    amkeyte

    Morning!

    So I'm looking at th problem of recreating the Location instance for retrieval out of the LocationData class. In WorldData it requests a server for getWorld, and that makes sense for a world. In order to wrap it, however I need a reference to server, and a function that is getLocation(server) seems a little off to me. In fact I could see this coming up in a lot of places, getChunk(server) or getVehicle(server) etc.. I'm curious if adding an accessor to the Permissions singleton to server to allow persisted data members to tie back into the "real world" by name (such as the way you've done in getWorld would be a good idea?


    Code:
     private Persistence persistence = Persistence.getInstance();
    
    ...
    
        public Location getLocation() {
            if (location != null) {
                return location;
            }
    
            World world = worldData.getWorld(persistence.getServer());
    
            return new Location(
                    world,
                    blockVector.getX(),
                    blockVector.getY(),
                    blockVector.getZ(),
                    orientation.getYaw(),
                    orientation.getYaw());
        }
    I think that since these classes are specifically for use with Persistence that it would be acceptable to load the singleton here. Else we have to look at somehow using a LocationData.setServer(server) or again, getLocation(server), which seems clunky to me.

    Thoughts?
     
  4. Offline

    NathanWolf

    Right! Great idea- I'd like a getLocation(Server), a getLocation(World), and a getLocation(WorldData), really, ideally :) But whichever ones you need to use, go ahead and feel free to implement.

    Any classes that will be used specifically as Persistence DAO's are safe to access the singleton anywhere in there- so you can look up references to WorldData based on a world name, for instance, if you need to - but I imagine getLocation(World) probably covers the most basic use case- you always need a World when creating a Location anyway.

    Thanks for doing this! :D
    --- merged: Feb 13, 2011 6:43 PM ---
    Oh, also- i didn't notice the "by name" part- you can do that, too, but it'll be like a one-line accessor since you can lookup a WorldData by a name :)

    So, yeah, feel free to add getLocation(String worldName) to the list, too, if you think it's helpul.

    My feeling is, you can never have too many different function signatures, as long as they all make sense and handle a use case in at least a slightly more efficient or simple way than one of the others. Just my opinion, though, of course :)
     
  5. Offline

    amkeyte

    Hmm I'll stop hijacking your thread on this :) creating an issue on your github instead
     
  6. Offline

    NathanWolf

    Great idea!

    Though, this really is a great place for discussion like this- but I think at this point we can pretty much say "it'll be in the API later" and handle the details in the tracker...
    --- merged: Feb 13, 2011 6:49 PM ---
    I'm very interested, by the way, because I think I might start using it, like, immediately in NetherGate :)

    I'm currently storing Portals as a WorldData/BoundingBox, I was going to change that to a WorldData/BlockVector/BlockFace for efficiency in a near-term refactor to start actually using my Portal object :)

    Anyway, I really only need a "bit" of orientation info (literally a bit- it's either NS or EW facing) - so a BlockFace was the closest I could get. But, heck, why not store full-on orientation and just use a persistable LocationData? Who knows, maybe I can support some sort of crazy-angled portal one day? ;)
    --- merged: Feb 13, 2011 10:49 PM ---
    Hey, just FYI- I got a tip in IRC about Doxygen, which is what they use to generate the Bukkit javadocs.

    Long story short, the Persistence Documentation is now much prettier!

    I think there's a lot I can do with this in the future, especially the main page, in terms of linking to example code and introducing Persistence- so that may become the go-to place for Persistence eventually (until Fill is ready, or whatever).
     
  7. I get massive lag while using this as well, particularly at the "Downloading Terrain" screen. Takes about 10 times longer than usual with this mod. Either this, or NetherGate which uses this, is causing it.
     
  8. Offline

    NathanWolf

    That would be Nethergate, most likely :)

    Creating extra worlds will definitely lag things. After that, it should settle down.

    It's mainly the initial creation of a world, or new tiles. Nether worlds are "worse", from what I've seen (and can imagine), performance-wise. First load of a world will take a bit, too, to generate the spawn area just like at server startup

    There is a cost associated with each additional world loaded there at runtime, too- but it doesn't seem too bad. I've got at least 4 loaded at a time, generally, on my public server.

    Check the NetherGate thread for more info, but I also understand there to be some sort of multi-world bug that causes entities to accumulate, and I've certainly experienced this myself, as well as experienced lag from it. There's a "nuke" command to take care of this.

    I don't think this is a Persistence issue, but it is something that keeps coming up. I'm going to add some debug logging so we can tell what Persistence is really doing.

    The "Downloading Terrain" bit is the most credible piece of evidence yet, since Persistence does save on player join- but still... I don't think that much data should be changing at this point for that to cause any sort of delay, unless someone is doing something crazy with Persistence I don't know about... I'm definitely doing nothing that I think should cause lag in NetherGate, for instance :) [er, beyond, you know, loading other worlds and stuff...]
    --- merged: Feb 14, 2011 1:03 AM ---
    I seem to remember a conversation in here a while ago about when it's ok to use Persistence.getInstance().

    I've checked the code, I think the answer is actually "anywhere" :)

    The plugin does nothing to really initialize Persistence other than to just get things going..
     
  9. Offline

    amkeyte

    Alright! the LocationData class is here:

    http://code.google.com/p/amkeyte-bu...kit/plugins/persistence/dao/LocationData.java

    This is, at this time, untested code as my server is hosed and not bootable due to other issues.

    For the most part you should be able to instantiate and use this class as you would a normal Location in your own code. It has most of the methods used in Location (with a couple slight modifications) and also matching constructors. You can also just construct by inputting a Location Class, or nothing at all, then using setters for the blockVector, worldData, and orientation fields.

    Use the locationFactory method to get an instance of Location with the persisted data. This method just spits out a whole new instance, and if you change any of its fields it will not affect the data stored in its associated LocationData.
    hence my choice not to use "getLocation". it would be too tempting for someone to try something like:

    Code:
    //set x using the Location instance
    myLocationData.getLocation().setX(123);
    //and then expect this to work:
    mylocationData.getX(); // == 123? NO!! 
    hopefully the Factory name will add clarity to that fact. I had to do this due to complications with keeping all the data synchronized within the LocationData class.

    Currently there is no method locationFactory(world) because I am unable to get any internal access to the server instance. (tried looking around in persistence and pluginUtilities but didn't see an obvious way to do it.) As you are aware using WorldData requires this (WorldData.getWorld(server)) and there is no reference to the server in the World class. So, this brings us back to the "can we get accessors to bukkit vars inside DAO's" that we are discussing on your github.

    so there ya have it. Please review this for problems and please let me know what you think. I'm still getting the hang of Java.. lol
     
  10. Offline

    NathanWolf

    This is great! Thanks!

    I want to get this in as a core class ASAP, assuming you don't mind. I'll want to edit it up, probably, but I want to keep your name on the @author tag for sure! That tag may end up in Bukkit core one day ;)

    So, as for "factory"- I'd like to discuss that. I see where you're coming from, and it may not be best practice but I use "get" in a few places where it isn't necessarily a "getter"- like you're getting the property of an object. For instance, WorldData.getWorld(Server)- not a great example, since it's not creating anything, I guess...

    But I think I'd prefer them to just be "getLocation"- I'm open for debate though :)

    A "factory", to me, is more something meant for instantiating different types of objects based on some sort of criteria- at least, in my mind.

    Also, generally with persistence, it's a bad idea to do things like

    Code:
    MyObject.getLocation().setX(0);
    That won't persist the way you want, since it doesn't know that you've modified "location", even if you put() MyObject.

    It's also just not a great idea in general- you should always get() into a local variable and check for null anyway, really :)

    There are a couple of other minor things- I don't think any of those fields need to be readonly, for instance- that's really only reserved for exceptional cases, like when your id is a hash function (such as BlockVector).

    So, yeah, other than the factory/accessor name thing (and, again, we can discuss) I'm ready to take this thing more or less wholesale!

    Thanks again!
    --- merged: Feb 14, 2011 4:35 PM ---
    Ok, @Raphfrk has got me real excited about BukkitScheduler now, and I want some feedback from interested parties.

    I already knew that I was going to use BukkitSched for incremental auto-save, this just makes good sense.

    What I did not realize, because I hadn't had a chance to look into Raph's tutorial yet, is that I think he can actually give us one better - asynchronous incremental auto-save!

    Actually, heck, no real reason to bother making it incremental if I can make it asynch! :p That was just to avoid lagging things during periods of heavy data modification, spread the load out- but who cares? I get my own thread!! (EDIT: hm, maybe still worth doing, since a write during an auto-save will still lock the main thread... best to keep the actual saves as short and atomic as possible- maybe just start by only do one instance at a time- a single instance can actually be pretty massive, depending on whether it's got any lists....)

    So, I need to think about this, a lot. There are clearly some locking issues I need to work out, but I know that already and have an issue for it. Persistence is really over-aggressive right now about thread safety- for asynch to be worthwhile, I need to allow read access during a save- so concurrent reads, but be able to lock out writes- and do all this in a safe way, of course.

    But, I think, once the internals are worked out, from an external perspective I've only been able to think of one ramification so far- and this is where I need your feedback.

    You may not call Bukkit API functions from the getter of a DAO!

    Can we live with that? I think we can- that seems like a really weird thing to do, to begin with.

    Note that I only care about real, PersistField getters. So, if you've got a LocationData.getWorld(Server), this is not what I mean- just something I'm going to touch when I walk the cache to save data- I can't call your getter from another thread if your getter is then going to call Bukkit API- that would be bad.

    So, let me know what you think - if I don't get any negative feedback, I'm going to go ahead and push forward with this, because I think it'll be just about sweeter than molasses, and should finally put to bed any question of Persistence lagging peoples' servers :p

    BTW, the next released version of Persistence will have some logging. I plan to let you turn this off once there's a properties file, but for now I'm hoping it's not too spammy. If it's spamming your logs like crazy, there's probably something really wrong!

    It should tell you how many rows it reads/writes now- that's the main thing. Knowing when and how much data is being saved/loaded should give us a lot more visibility into what Persistence is doing behind the scenes.

    Thanks!
     
  11. Offline

    amkeyte

    Please use this code as you see fit. It wouldn't exist without your groundwork.

    the readonly thing is definitely an "appendix" that needs to be "operated on". I will pull it out in the next version (available late tonight)

    On the Factory issue, I agree, I would rather call it something else, and sort of anticipated this conversation :)

    Your example above is the EXACT reason I want to avoid using getLocation in this specific case. As you can see in my code example above, that's what I was trying to convey - though perhaps poorly. Changes in that generated Location instance will not be propagated back to LocationData. What about generateLocation() or something else that denotes that the location instance is being generated back out of persisted data. Again, I agree that Factory is inappropriate. I think this conversation is worth having in order to define a standard to guide future cases.

    some brainstorming:
    generateLocation(server)
    createLocation(server)
    buildLocation(server)
    newLocation(server) -- don't like this one. returned Location is not default
    generateLocation(server)
    initLocation(server) -- bad. implies that all location data will be reset (whatever that would mean lol)
    copyLocation(server) -- might be good... not quite "clone" but more than "new"
    dittoLocation(server) -- it's "cute", maybe not quite right though...

    actually, (ok so i'm looking through the thesaurus for "copy" now... bear with me lol) how about this:

    Code:
    public Location imprintLocation(Location location){
    
    ....
    match the world name or throw an error (or possibly return null?)
    
    set all the location variables
    
    ...
    
       return location;
    
    This method has a few advantages. First, the client code is supplying the actual instance of the object to be "imprinted" which relieves the method of actually having to generate one. This automatically eliminates my concern that the client coder may interpret this object as persistable. Also there is no more need to pass in a server instance, since the world is set inside the location parameter before our method even has to deal with it. We just check to make sure its the correct world, then set the persisted values, and return the same instance right back to the client. They know where it came from, and they know how it's values will be handled.

    Imprint is a good word (if a little long) for this because it implies that there is an object that is being put in the machine, and the machine stamps it data onto the object. I would guess that there is an established OOP term for this type of action, one that I am unaware of. Please feel free to educate me on this! :)

    So, whatever case, I think this is the method we should use to put persisted data into actual bukkit objects: let the client code generate the object, and we set the values.

    I'll include an example in my next iteration of LocationData.


    As far as multi threaded applications, I will be unable to provide an educated opinion. In fact, I think some examples are in order for:

    If nothing else at least for my clarity! lol

    Ok... its wife time.. I'll work on this tonight!
     
  12. Offline

    NathanWolf

    Ok, I got super smalled at work today, so this took may too long considering all I did was put it in and re-publish (and comment out some in-progress data migration code...)

    0.39 has LocationData, courtesy of Amekyte!

    This is a great DAO to use for persisting a multi-world-compliant location. It uses a BlockVector, so you can hash by worldname and then location, if you want to easily cache LocationData objects.

    It's also got an Orientation, which is something I threw together to encapsulate pitch/yaw- not really the way Bukkit does it internally, but I like it (and if it was good enough for Unreal Engine 3, it's good enough for me...).

    This means that, using a Server instance, you can get a full-fledged Location back from a LocationData.

    Really, this is just perfect, and something I, personally, need for NetherGate immediately. I hope you all find it useful, too!

    Thanks again, @amkeyte!
    --- merged: Feb 15, 2011 4:03 AM ---
    Heh.... prolly shoulda read all this before I pushed out 0.39 ;)

    If you're up for it, feel free to fork BukkitPlugins and request a pull on LocationData, copy+paste your new version in. If that's too much trouble (seriously, it can be) I can re-integrate your latest.

    I want to look this over, too, make sure there's nothing missing (at least in terms of what I need for NetherGate).

    And then there's the whole generator/factory/get thing- I totally agree, we need to define a precedent. I already have, sort of, but it's with "get" and I can see potential confusion- though I am willing to argue for it if I don't find a suitable replacement.

    Hmm... can we agree to use "get" for now, and postpone the larger decision? It honestly may be something we want to ask the Bukkit devs, they might have an opinion, or maybe be aware of a best practice that I'm not.
    --- merged: Feb 15, 2011 4:35 AM ---
    Ok, rather then bug the devs, I did the legwork and looked around a bit in the Bukkit code for examples of this kinda thing... seems like maybe "getNewXXX" is the winner?

    e.g.:


    Code:
    /**
         * Constructs a new MaterialData relevant for this Material, with the given
         * initial data
         *
         * @param raw Initial data to construct the MaterialData with
         * @return New MaterialData with the given data
         */
        public MaterialData getNewData(final byte raw) {
            if (data == null) {
                return null;
            }
    ...
    And, then, maybe for the "String" World-by-name getter of WorldData, for instance, we could use "match" like this:

    Code:
     public static Material matchMaterial(final String name) {
            Material result = null;
    
            try {
                result = getMaterial(Integer.parseInt(name));
            } catch (NumberFormatException ex) {
            }
    
            if (result == null) {
                String filtered = name.toUpperCase();
                filtered = filtered.replaceAll("\\s+", "_").replaceAll("\\W", "");
                result = lookupName.get(filtered);
            }
    
            return result;
        }

    Oh, and there's also "to":

    Code:
        /**
         * Creates a new ItemStack based on this MaterialData
         *
         * @return New ItemStack containing a copy of this MaterialData
         */
        public ItemStack toItemStack() {
            return new ItemStack(type);
        }
    So.. hm.. yeah, I dunno :)

    I think I can follow this as a template, if you want to hand over the keys once you're "done" with the code, and you don't mind me changing some function names up a bit down the road- feel free to try to get it as close to Bukkit standard as possible if you want. You're of course always willing to do the minimum work required here to get your stuff going, and I'll take it from there- whatever you wanna do :)

    Clearly my "get" genericness is not the way to go, though, so I'm glad I looked around :)
     
  13. Offline

    amkeyte

    okees... LocationData is added on gitHub and pull request is pending. Included all the changes we discussed and fixed a bug I found in 0.39 version. (my bad! >.<)

    went with toLocation(server) and added toLocation(location). Obviously I like "to". the perfect and obvious fit imo.

    I have attempted the pull request, but I'm a little clueless there, hope it worked right. If not a little coaching for the future, please, and for now you can copy and paste the file from my link above.
     
  14. Offline

    NathanWolf

    I always have trouble will pull requests, too! It always says there's ac conflict when they're shouldn't be ... I'll get it pulled in tomorrow, maybe double-check it, and get a new release in order. Worse-case I can copy+paste the code again :D

    I think I may have a bug of my own- I saw some strange behavior earlier where "sender" data objects started collecting- I had like 1000 of them in my global.sender table... really weird. I couldn't get it to happen again, so I'm hoping it was some strange testing thing, but it made me nervous :(
    --- merged: Feb 15, 2011 8:13 PM ---

    BoundingBox

    I wanted to post this conversation I had about BoundingBox- you may not know it's in there, but if you're doing any sort of construction or world-inspection work, it may make good sense to be using it!

    BoundingBox is just a (persistable) wrapper for two BlockVectors, but it's got some really handy helper functions:

    fill(Material, data, HashMap<>): fills an area with the specified material type. Takes an exclusion list, so you can make sure to avoid overwriting important blocks like chests and signs.

    getBlocks(World, List): Populates an x,y,z ordered list of blocks- use this to query for blocks from a (loaded) world.

    getFace(BlockFace, thickness, offset): Create a new BoundingBox from this one, given a thickness and offset (from the outside of the original bounding box). This is great for quickly constructing walls, ceiling, floor, etc. (@amkeyte... another function that needs a better name! ;))

    I had Nethergate building a gigantic bedrock box with this code using getFace(), fill(), and a clever (if i do say so myself) "for" loop with an array of all 6 BlockFace values :) It's a single line of code! (technically...)

    Anyway, I think that's about it. I do want to mention that BoundingBox will be migrating soon to a Gameplay plugin that I'm working on. I'll probably leave it in Persistence for a revision or two (deprecate it) until you've had a chance to migrate, but it really doesn't belong in Persistence, per se.
     
  15. Offline

    phaed

    How can I stop this new version from spamming the console with these debug messages "saved 29 objects to global.player". The amount of these on my console now is overwhelming and have rather little use.
     
  16. Offline

    NathanWolf

    Hm- it really shouldn't be saving that many player at one time :(

    I'll turn the debugging logging off until I've had a chance to make a properties file option for it. Meantime, I need to investigate that- I think players should get saved one at a time (on login/logout), unless you're modifying and put()'ing PlayerData instances yourself.
    --- merged: Feb 16, 2011 1:32 AM ---
    0.40 Released

    This turns off all that logging for now- there will be some advanced logging features in the properties file, once there is one.

    This release also adds a Permissions node (Persistence.commands.su) for the "su" command so admins can disable its use. I thought people trusted their ops unconditionally, but that's really not the case. This is why we have things like the sudoers file, and the new pnode takes care of that nicely.

    I'd like to add an option so that you can specify whether or not your plugin respects "su", in case you want to use the Persistence wrappers for Permissions, but don't like the idea of "su". That's a bit more complex, though, so it might be a few releases away- I have bigger fish to fry, unless it's a make-or-break for someone.

    Ok, automatic data migration (phase 1, simple form) is on its way, hopefully soon. I hope that's not holding up progress for anyone....
    --- merged: Feb 16, 2011 1:32 AM ---
    Oh, this release is also (I think) properly Maven-ized. Thanks to @WinSock for his copious help with that!
     
  17. Offline

    phaed

    Check it

    Code:
    2011-02-15 19:17:51 [INFO] Persistence: loaded 2 objects from global.vector
    2011-02-15 19:18:00 [INFO] Persistence: saved 2 objects to global.plugin
    2011-02-15 19:18:00 [INFO] Persistence: saved 4 objects to global.pluginCommands
    2011-02-15 19:18:00 [INFO] Persistence: saved 4 objects to global.pluginCommands
    2011-02-15 19:18:00 [INFO] Persistence: saved 38 objects to global.pluginMessages
    2011-02-15 19:18:00 [INFO] Persistence: saved 38 objects to global.pluginMessages
    2011-02-15 19:18:00 [INFO] Persistence: saved 24 objects to global.command
    2011-02-15 19:18:00 [INFO] Persistence: saved 19 objects to global.commandUsage
    2011-02-15 19:18:00 [INFO] Persistence: saved 19 objects to global.commandUsage
    2011-02-15 19:18:00 [INFO] Persistence: saved 17 objects to global.commandSenders
    2011-02-15 19:18:00 [INFO] Persistence: saved 17 objects to global.commandSenders
    2011-02-15 19:18:01 [INFO] Persistence: saved 38 objects to global.message
    2011-02-15 19:18:01 [INFO] Persistence: saved 17 objects to global.player
    2011-02-15 19:18:01 [INFO] Persistence: saved 17 objects to nether.player
    2011-02-15 19:18:09 [INFO] Persistence: saved 9 objects to global.player
    2011-02-15 19:18:09 [INFO] Persistence: saved 6 objects to nether.player
    2011-02-15 19:18:33 [INFO] Persistence: saved 18 objects to global.player
    2011-02-15 19:18:33 [INFO] Persistence: saved 17 objects to nether.player
    2011-02-15 19:19:02 [INFO] Persistence: saved 20 objects to global.player
    2011-02-15 19:19:02 [INFO] Persistence: saved 19 objects to nether.player
    2011-02-15 19:19:57 [INFO] Persistence: saved 21 objects to global.player
    2011-02-15 19:19:57 [INFO] Persistence: saved 20 objects to nether.player
    2011-02-15 19:20:14 [INFO] Persistence: saved 19 objects to global.player
    2011-02-15 19:20:14 [INFO] Persistence: saved 18 objects to nether.player
    2011-02-15 19:20:18 [INFO] Persistence: saved 16 objects to global.player
    2011-02-15 19:20:18 [INFO] Persistence: saved 12 objects to nether.player
    2011-02-15 19:21:16 [INFO] Persistence: saved 21 objects to global.player
    2011-02-15 19:21:16 [INFO] Persistence: saved 20 objects to nether.player
    2011-02-15 19:22:27 [INFO] Persistence: saved 21 objects to global.player
    2011-02-15 19:22:32 [INFO] Persistence: saved 21 objects to nether.player
    2011-02-15 19:22:43 [INFO] Persistence: saved 18 objects to global.player
    2011-02-15 19:22:43 [INFO] Persistence: saved 17 objects to nether.player
    2011-02-15 19:23:13 [INFO] Persistence: saved 19 objects to global.player
    2011-02-15 19:23:13 [INFO] Persistence: saved 18 objects to nether.player
    2011-02-15 19:23:25 [INFO] Persistence: saved 17 objects to global.player
    2011-02-15 19:23:25 [INFO] Persistence: saved 15 objects to nether.player
    2011-02-15 19:23:48 [INFO] Persistence: saved 17 objects to global.player
    2011-02-15 19:23:48 [INFO] Persistence: saved 16 objects to nether.player
    2011-02-15 19:24:28 [INFO] Persistence: saved 19 objects to global.player
    2011-02-15 19:24:28 [INFO] Persistence: saved 18 objects to nether.player
    2011-02-15 19:24:46 [INFO] Persistence: saved 19 objects to global.player
    2011-02-15 19:24:47 [INFO] Persistence: saved 18 objects to nether.player
    2011-02-15 19:25:05 [INFO] Persistence: saved 19 objects to global.player
    2011-02-15 19:25:05 [INFO] Persistence: saved 19 objects to nether.player
    2011-02-15 19:25:06 [INFO] Persistence: saved 15 objects to global.player
    2011-02-15 19:25:06 [INFO] Persistence: saved 12 objects to nether.player
    2011-02-15 19:25:17 [INFO] Persistence: saved 11 objects to global.player
    2011-02-15 19:25:17 [INFO] Persistence: saved 7 objects to nether.player
    2011-02-15 19:25:19 [INFO] Persistence: saved 17 objects to global.player
    2011-02-15 19:25:19 [INFO] Persistence: saved 16 objects to nether.player
    2011-02-15 19:26:57 [INFO] Persistence: saved 22 objects to global.player
    2011-02-15 19:26:58 [INFO] Persistence: saved 22 objects to nether.player
    2011-02-15 19:26:58 [INFO] Persistence: loaded 2 objects from global.vector
    2011-02-15 19:27:34 [INFO] Persistence: saved 2 objects to global.plugin
    2011-02-15 19:27:34 [INFO] Persistence: saved 4 objects to global.pluginCommands
    2011-02-15 19:27:34 [INFO] Persistence: saved 4 objects to global.pluginCommands
    2011-02-15 19:27:34 [INFO] Persistence: saved 38 objects to global.pluginMessages
    2011-02-15 19:27:34 [INFO] Persistence: saved 38 objects to global.pluginMessages
    2011-02-15 19:27:34 [INFO] Persistence: saved 24 objects to global.command
    2011-02-15 19:27:35 [INFO] Persistence: saved 19 objects to global.commandUsage
    2011-02-15 19:27:35 [INFO] Persistence: saved 19 objects to global.commandUsage
    2011-02-15 19:27:35 [INFO] Persistence: saved 17 objects to global.commandSenders
    2011-02-15 19:27:35 [INFO] Persistence: saved 17 objects to global.commandSenders
    2011-02-15 19:27:35 [INFO] Persistence: saved 38 objects to global.message
    2011-02-15 19:27:35 [INFO] Persistence: saved 20 objects to global.player
    2011-02-15 19:27:35 [INFO] Persistence: saved 20 objects to nether.player
    2011-02-15 19:28:55 [INFO] Persistence: saved 21 objects to global.player
    2011-02-15 19:28:55 [INFO] Persistence: saved 20 objects to nether.player
    2011-02-15 19:28:56 [INFO] Persistence: saved 20 objects to global.player
    2011-02-15 19:28:56 [INFO] Persistence: saved 18 objects to nether.player
    2011-02-15 19:30:21 [INFO] Persistence: saved 20 objects to global.player
    2011-02-15 19:30:21 [INFO] Persistence: saved 20 objects to nether.player
    2011-02-15 19:30:22 [INFO] Persistence: saved 16 objects to global.player
    2011-02-15 19:30:22 [INFO] Persistence: saved 16 objects to nether.player
    2011-02-15 19:30:41 [INFO] Persistence: saved 18 objects to global.player
    2011-02-15 19:30:41 [INFO] Persistence: saved 17 objects to nether.player
    2011-02-15 19:30:45 [INFO] Persistence: saved 16 objects to global.player
    2011-02-15 19:30:45 [INFO] Persistence: saved 14 objects to nether.player
    2011-02-15 19:30:55 [INFO] Persistence: saved 17 objects to global.player
    2011-02-15 19:30:55 [INFO] Persistence: saved 17 objects to nether.player
    2011-02-15 19:31:03 [INFO] Persistence: saved 16 objects to global.player
    2011-02-15 19:31:03 [INFO] Persistence: saved 14 objects to nether.player
    2011-02-15 19:31:04 [INFO] Persistence: saved 16 objects to global.player
    2011-02-15 19:31:04 [INFO] Persistence: saved 14 objects to nether.player
    2011-02-15 19:31:12 [INFO] Persistence: saved 15 objects to global.player
    2011-02-15 19:31:12 [INFO] Persistence: saved 10 objects to nether.player
    2011-02-15 19:31:28 [INFO] Persistence: saved 17 objects to global.player
    2011-02-15 19:31:28 [INFO] Persistence: saved 16 objects to nether.player
    2011-02-15 19:31:31 [INFO] Persistence: saved 17 objects to global.player
    2011-02-15 19:31:32 [INFO] Persistence: saved 17 objects to nether.player
    2011-02-15 19:31:40 [INFO] Persistence: saved 16 objects to global.player
    2011-02-15 19:31:40 [INFO] Persistence: saved 15 objects to nether.player
    2011-02-15 19:31:41 [INFO] Persistence: saved 15 objects to global.player
    2011-02-15 19:31:41 [INFO] Persistence: saved 11 objects to nether.player
    2011-02-15 19:31:44 [INFO] Persistence: saved 14 objects to global.player
    2011-02-15 19:31:44 [INFO] Persistence: saved 13 objects to nether.player
    2011-02-15 19:31:50 [INFO] Persistence: saved 13 objects to global.player
    2011-02-15 19:31:50 [INFO] Persistence: saved 11 objects to nether.player
    2011-02-15 19:32:02 [INFO] Persistence: saved 15 objects to global.player
    2011-02-15 19:32:02 [INFO] Persistence: saved 13 objects to nether.player
    2011-02-15 19:32:05 [INFO] Persistence: saved 17 objects to global.player
    2011-02-15 19:32:05 [INFO] Persistence: saved 17 objects to nether.player
    2011-02-15 19:32:15 [INFO] Persistence: saved 15 objects to global.player
    2011-02-15 19:32:15 [INFO] Persistence: saved 13 objects to nether.player
    2011-02-15 19:32:35 [INFO] Persistence: saved 17 objects to global.player
    2011-02-15 19:32:35 [INFO] Persistence: saved 16 objects to nether.player
    2011-02-15 19:32:42 [INFO] Persistence: saved 15 objects to global.player
    2011-02-15 19:32:42 [INFO] Persistence: saved 13 objects to nether.player
    2011-02-15 19:32:45 [INFO] Persistence: saved 16 objects to global.player
    2011-02-15 19:32:45 [INFO] Persistence: saved 14 objects to nether.player
    2011-02-15 19:32:59 [INFO] Persistence: saved 18 objects to global.player
    2011-02-15 19:32:59 [INFO] Persistence: saved 17 objects to nether.player
    2011-02-15 19:33:07 [INFO] Persistence: saved 17 objects to global.player
    2011-02-15 19:33:07 [INFO] Persistence: saved 15 objects to nether.player
    2011-02-15 19:34:49 [INFO] Persistence: saved 21 objects to global.player
    2011-02-15 19:34:49 [INFO] Persistence: saved 21 objects to nether.player
    2011-02-15 19:34:57 [INFO] Persistence: saved 18 objects to global.player
    2011-02-15 19:34:58 [INFO] Persistence: saved 17 objects to nether.player
    2011-02-15 19:35:41 [INFO] Persistence: saved 20 objects to global.player
    2011-02-15 19:35:41 [INFO] Persistence: saved 20 objects to nether.player
    2011-02-15 19:36:50 [INFO] Persistence: saved 21 objects to global.player
    2011-02-15 19:36:50 [INFO] Persistence: saved 20 objects to nether.player
    2011-02-15 19:36:55 [INFO] Persistence: saved 16 objects to global.player
    2011-02-15 19:36:55 [INFO] Persistence: saved 15 objects to nether.player
    2011-02-15 19:37:02 [INFO] Persistence: saved 8 objects to global.player
    2011-02-15 19:37:02 [INFO] Persistence: saved 7 objects to nether.player
    2011-02-15 19:37:46 [INFO] Persistence: saved 20 objects to global.player
    2011-02-15 19:37:46 [INFO] Persistence: saved 20 objects to nether.player
    2011-02-15 19:37:54 [INFO] Persistence: saved 19 objects to global.player
    2011-02-15 19:37:55 [INFO] Persistence: saved 16 objects to nether.player
    2011-02-15 19:38:00 [INFO] Persistence: saved 17 objects to global.player
    2011-02-15 19:38:00 [INFO] Persistence: saved 16 objects to nether.player
    2011-02-15 19:38:15 [INFO] Persistence: saved 19 objects to global.player
    2011-02-15 19:38:15 [INFO] Persistence: saved 17 objects to nether.player
    2011-02-15 19:38:29 [INFO] Persistence: saved 19 objects to global.player
    2011-02-15 19:38:29 [INFO] Persistence: saved 19 objects to nether.player
    2011-02-15 19:38:37 [INFO] Persistence: saved 18 objects to global.player
    2011-02-15 19:38:37 [INFO] Persistence: saved 15 objects to nether.player
    2011-02-15 19:39:08 [INFO] Persistence: saved 21 objects to global.player
    2011-02-15 19:39:08 [INFO] Persistence: saved 20 objects to nether.player
    2011-02-15 19:40:24 [INFO] Persistence: saved 20 objects to global.player
    2011-02-15 19:40:24 [INFO] Persistence: saved 20 objects to nether.player
    2011-02-15 19:42:27 [INFO] Persistence: saved 19 objects to global.player
    2011-02-15 19:42:27 [INFO] Persistence: saved 19 objects to nether.player
    2011-02-15 19:42:36 [INFO] Persistence: saved 15 objects to global.player
    2011-02-15 19:42:36 [INFO] Persistence: saved 13 objects to nether.player
    2011-02-15 19:42:42 [INFO] Persistence: saved 15 objects to global.player
    2011-02-15 19:42:42 [INFO] Persistence: saved 14 objects to nether.player
    2011-02-15 19:42:50 [INFO] Persistence: saved 16 objects to global.player
    2011-02-15 19:42:50 [INFO] Persistence: saved 15 objects to nether.player
    2011-02-15 19:44:07 [INFO] Persistence: saved 19 objects to global.player
    2011-02-15 19:44:07 [INFO] Persistence: saved 19 objects to nether.player
    2011-02-15 19:44:28 [INFO] Persistence: saved 19 objects to global.player
    2011-02-15 19:44:29 [INFO] Persistence: saved 18 objects to nether.player
    2011-02-15 19:46:36 [INFO] Persistence: saved 20 objects to global.player
    2011-02-15 19:46:36 [INFO] Persistence: saved 19 objects to nether.player
    2011-02-15 19:46:49 [INFO] Persistence: saved 18 objects to global.player
    2011-02-15 19:46:49 [INFO] Persistence: saved 18 objects to nether.player
    2011-02-15 19:46:57 [INFO] Persistence: saved 16 objects to global.player
    2011-02-15 19:46:58 [INFO] Persistence: saved 15 objects to nether.player
    2011-02-15 19:47:18 [INFO] Persistence: saved 17 objects to global.player
    2011-02-15 19:47:18 [INFO] Persistence: saved 17 objects to nether.player
    2011-02-15 19:47:25 [INFO] Persistence: saved 17 objects to global.player
    2011-02-15 19:47:25 [INFO] Persistence: saved 16 objects to nether.player
    2011-02-15 19:47:54 [INFO] Persistence: saved 19 objects to global.player
    2011-02-15 19:47:54 [INFO] Persistence: saved 18 objects to nether.player
    2011-02-15 19:48:16 [INFO] Persistence: saved 19 objects to global.player
    2011-02-15 19:48:16 [INFO] Persistence: saved 17 objects to nether.player
    2011-02-15 19:48:36 [INFO] Persistence: saved 18 objects to global.player
    2011-02-15 19:48:36 [INFO] Persistence: saved 18 objects to nether.player
    2011-02-15 19:48:40 [INFO] Persistence: saved 16 objects to global.player
    2011-02-15 19:48:40 [INFO] Persistence: saved 16 objects to nether.player
    2011-02-15 19:48:48 [INFO] Persistence: saved 14 objects to global.player
    2011-02-15 19:48:48 [INFO] Persistence: saved 13 objects to nether.player
    2011-02-15 19:48:50 [INFO] Persistence: saved 16 objects to global.player
    2011-02-15 19:48:50 [INFO] Persistence: saved 14 objects to nether.player
    2011-02-15 19:49:13 [INFO] Persistence: saved 20 objects to global.player
    2011-02-15 19:49:13 [INFO] Persistence: saved 18 objects to nether.player
    2011-02-15 19:50:15 [INFO] Persistence: saved 21 objects to global.player
    2011-02-15 19:50:15 [INFO] Persistence: saved 20 objects to nether.player
    2011-02-15 19:50:17 [INFO] Persistence: saved 17 objects to global.player
    2011-02-15 19:50:17 [INFO] Persistence: saved 13 objects to nether.player
    2011-02-15 19:50:30 [INFO] Persistence: saved 19 objects to global.player
    2011-02-15 19:50:30 [INFO] Persistence: saved 18 objects to nether.player
    2011-02-15 19:50:47 [INFO] Persistence: saved 20 objects to global.player
    2011-02-15 19:50:47 [INFO] Persistence: saved 19 objects to nether.player
    2011-02-15 19:50:54 [INFO] Persistence: saved 19 objects to global.player
    2011-02-15 19:50:54 [INFO] Persistence: saved 17 objects to nether.player
    2011-02-15 19:51:13 [INFO] Persistence: saved 20 objects to global.player
    2011-02-15 19:51:13 [INFO] Persistence: saved 18 objects to nether.player
    2011-02-15 19:51:22 [INFO] Persistence: saved 18 objects to global.player
    2011-02-15 19:51:22 [INFO] Persistence: saved 16 objects to nether.player
    2011-02-15 19:51:59 [INFO] Persistence: saved 21 objects to global.player
    2011-02-15 19:51:59 [INFO] Persistence: saved 21 objects to nether.player
    2011-02-15 19:52:07 [INFO] Persistence: saved 19 objects to global.player
    2011-02-15 19:52:07 [INFO] Persistence: saved 17 objects to nether.player
    2011-02-15 19:52:12 [INFO] Persistence: saved 21 objects to global.player
    2011-02-15 19:52:12 [INFO] Persistence: saved 17 objects to nether.player
    2011-02-15 19:52:48 [INFO] Persistence: saved 23 objects to global.player
    2011-02-15 19:52:48 [INFO] Persistence: saved 22 objects to nether.player
    2011-02-15 19:52:54 [INFO] Persistence: saved 22 objects to global.player
    2011-02-15 19:52:54 [INFO] Persistence: saved 18 objects to nether.player
    2011-02-15 19:53:09 [INFO] Persistence: saved 23 objects to global.player
    2011-02-15 19:53:09 [INFO] Persistence: saved 23 objects to nether.player
    2011-02-15 19:53:29 [INFO] Persistence: saved 24 objects to global.player
    2011-02-15 19:53:30 [INFO] Persistence: saved 22 objects to nether.player
    2011-02-15 19:54:01 [INFO] Persistence: saved 24 objects to global.player
    2011-02-15 19:54:01 [INFO] Persistence: saved 23 objects to nether.player
    2011-02-15 19:54:16 [INFO] Persistence: saved 22 objects to global.player
    2011-02-15 19:54:16 [INFO] Persistence: saved 21 objects to nether.player
    2011-02-15 19:54:28 [INFO] Persistence: saved 22 objects to global.player
    2011-02-15 19:54:28 [INFO] Persistence: saved 21 objects to nether.player
    2011-02-15 19:54:53 [INFO] Persistence: saved 22 objects to global.player
    2011-02-15 19:54:53 [INFO] Persistence: saved 22 objects to nether.player
    2011-02-15 19:55:00 [INFO] Persistence: saved 18 objects to global.player
    2011-02-15 19:55:00 [INFO] Persistence: saved 16 objects to nether.player
    2011-02-15 19:55:07 [INFO] Persistence: saved 19 objects to global.player
    2011-02-15 19:55:07 [INFO] Persistence: saved 19 objects to nether.player
    2011-02-15 19:55:13 [INFO] Persistence: saved 10 objects to global.player
    2011-02-15 19:55:13 [INFO] Persistence: saved 8 objects to nether.player
    2011-02-15 19:55:15 [INFO] Persistence: saved 17 objects to global.player
    2011-02-15 19:55:15 [INFO] Persistence: saved 17 objects to nether.player
    2011-02-15 19:55:23 [INFO] Persistence: saved 7 objects to global.player
    2011-02-15 19:55:23 [INFO] Persistence: saved 5 objects to nether.player
    2011-02-15 19:55:37 [INFO] Persistence: saved 17 objects to global.player
    2011-02-15 19:55:38 [INFO] Persistence: saved 16 objects to nether.player
    
     
  18. Offline

    NathanWolf


    Egad!

    Please tell me it's only doing that at startup, and not, like all the time?

    Otherwise, I may have to end up taking some of the blame for all those "lagging my server" claims that get hurled my way :(

    I also notice that, if anything, it's NetherGate that's doing it - so even if I'm not to blame in some core Persistence way, I'd then be to blame for Doing It Wrong in NetherGate, however I must be doing that. I know I put() the player changes pretty often, but save() should only happen on player login/logout. If I'm calling save() in playermove() within NetherGate, I'll need to smack myself in the head...

    Ok, thanks for all this info- feel free to grab the latest Persistence.jar straight from this thread and drop it in.

    It'll still be doing all that writing in the background... which is bad... but at least it'll be silent about it... :\

    Ok, going to backburner work on auto data migration until I figure out what's going on here. Will keep you all posted!
    --- merged: Feb 16, 2011 1:50 AM ---
    Also, @phaed, is it possible that all of these are correlated with player login/logout? How big is your server?

    Maybe that's not the best place for saving... I can't wait until I get BukkitScheduler in here for auto-save.

    In any case, the logging was clearly a bad idea outside of the context of debugging :) It's off now.
     
  19. Offline

    phaed

    Not that big, definetly dont have 100 people logging in/out a sec. Maybe someone logs in or out once evey 10-30 seconds.

    Constant, just like you see there, multple times a second, or every few seconds It's still happening as we speak. It is definetly lagging my server but people love the nether too much and are putting up with it.

    I can give you root on my linux box, you can ssh in and investigate yourself. Let me know.
     
  20. Offline

    NathanWolf

    @ssechaud

    This:

    Code:
    2011-02-16 14:24:27 [WARNING] Persistence: Error updating table pluginCommands: table pluginCommands has no column named commandId
    2011-02-16 14:24:27 [INFO] INSERT OR REPLACE INTO "pluginCommands" ("pluginId", "commandId") VALUES (?, ?)
    2011-02-16 14:24:27 [WARNING] Persistence: Error updating table pluginCommands: table pluginCommands has no column named commandId
    2011-02-16 14:24:27 [INFO] INSERT OR REPLACE INTO "pluginCommands" ("pluginId", "commandId") VALUES (?, ?)
    I *think* is from a bad data "migration" (using that very loosely here) form a previous version of Persistence.

    I mentioned this in the other thread, but basically that's core global data managed by Persistence - and (that one entity in particular, at least) is reasonably safe to delete.

    First, try the following... now, caveat: This is the first time I've ever tried this myself, so it's a bit funky ATM :)

    If you want to do this in-game, first use "/su" and make sure you get the "full access enabled" command. Otherwise, this can be done from the server console:

    Code:
    persist RESET foo global.plugin
    persist save
    
    Now couple of notes:
    1. At this point, if things we're working as they should be, we'd be done. However, in my testing, it does not re-create the tables on the fly- I need to look into why.
    2. The "foo" in the RESET command is necessary for now, but it's a bug :( An off-by-one error when I refactored PersistenceCommands to use Persistences' own command dispatch system, ironically enough :\
    Ok, now, you should be able to restart your server at this point and have everything work- let me know if not.

    Thanks for your patience - things like this will not happen nearly as much once the core global DAO's stabilize, and once data migration is there (though, honestly, I don't think auto migration would have been sophisticated enough to cover this ... :()

    My ultimate plan is to provide an easy system by which devs can roll out patch scripts, possibly embedded right in their jar files, for things that Persistence can't auto-migrate for you.

    Unfortunately, with any data storage solution, migration is always going to be an issue. A really big issue, in fact- but fear not, it's something I (painfully, painfully) deal with every day, supporting a huge code base of table schemas and stored procedures that is written for both MSSQL and Oracle (not written to support both - written twice, separately..... Systems can be put into place to make this easier on everyone, and hopefully transparent to admins.
    --- merged: Feb 16, 2011 5:57 PM ---
    Gak :(

    Shouldn't be necessary- I'll check this out right away... I swear, if NetherGate is calling save() in onPlayerMove, I'm going to make it protected right now. (And give myself a good smacking.... didn't I specifically tell people not to do that? Yeeesh...)

    Honestly, I'm kinda hoping that's it, though- something more "internal" would be bad- but I can't imagine what. Persistence doesn't even have a playermove listener, and I don't have auto save in yet ... :\

    Glad people love the nether! I'll get this fixed up for you- it's probably lagging my server, too :p Ole' betsy can't take this kind of stress....
    --- merged: Feb 16, 2011 6:15 PM ---
    Hm... tracking all the places I call save() anywhere in code...
    Code:
    onSU : this is the internal handler for the "/su" command.
    This persists your "SU" status so you can flip it on and leave it on.
    
    onSave : Handler for /persist save
    
    player login / join, Plugin.onDisable
    Nothing in NetherGate at all :(

    Now, I do recall, at one point, I was calling save() from NetherGate- when I tp'd a player.

    For some reason, it "seemed like a good idea at the time" to persist player's tp state. I have no idea why, looking back- it's actually a really stupid idea! :\

    Anyway, that would have only been on actual player portalling, and NetherGate hasn't done that in a few versions... but.... since you probably got your Persistence in the same zip as NetherGate, I'm going to assume you're up to date, and not try to pull that particular card.. :D

    Clearly this bears more investigation- I've got a 2PM deadline (Pacific US) today at work for something fairly critical... I'll try and look at this tonight, if I can- this could be really bad.

    Thanks for bringing this to my attention, @phaed!
    --- merged: Feb 16, 2011 8:45 PM ---

    Semi-Important Announcement

    Ugh... so.. I'm going to have to break global data again, in a similar way, for the next release- the bug I fixed with pluginCommands also exists in the Message DAO structure :(

    For details, since this was effectively a client error (though the client, in this case, is Persistences' internal PluginUtilities), here's about what happened:

    I have the "contained" tag on my PersistField in PluginData's Lists of Message and PluginCommand instances- the idea is to make them plugin-specific, and contain their data within the plugin data structure.

    Now, this is ultimately a refactoring issue- originally, messages and commands were not per-plugin, I was going to auto-prepend the plugin name to the string id to scope it (after toying with the idea of supporting multi-id entities.... bad idea!)- then I realized that was dumb, I could just store them within the plugin DAO itself as Lists, and that was probably better.

    Unfortunately, I got my wires crossed and put() the instances of Message and PluginCommand, as well as adding them to the contained List. Since they have id's (the id is meant to be unique within the context of a PluginData!), Persistence said "well, ok, sure" and persisted them for me.

    Since I wasn't using get() to check first, this was somehow causing rows to get added over and over again to a couple of tables.

    Now, this is where it gets hazy- there should be multiple levels of protection in place to keep things like this from happening, even with this kind of client error- not the least of which being the PK on the table, which from what I can tell was being violated all over the place. Maybe SqlLite doesn't even respect that? The PK definition is there, in the master table sql definition- so I'm not sure what's up with that.

    I also need to think about this particular situation, and if there is any way I can catch it and inform the dev- in this case, Message and PluginCommand could be used as stand-alone entities, but maybe they shouldn't be- in which case, I can add the "contained" tag and make sure I enforce it (as opposed to just allowing that entity to be id-less).

    I think this would resolve this particular issue, but there's always going to be the case where you could be containing something that's also OK to be a unique entity- and you don't want to mix and match the use of put() with the contained instances, in that case.

    It's all very complex, which is why, I think, I need to find a better way to make sure this doesn't happen to anyone else. Basically, if I'm able to accidentally use the API in such a way that causes this kind of trouble, then there's either something going wrong, or the API isn't enforced/consistent/clear/whatever enough :p

    Clearly, this bears more investigation- and perhaps an automatic way to clear global data by entity when they change, since generally (admin edits aside) all of this is data that can be recreated automatically anyway.

    I'll give it some thought and keep you posted.
    --- merged: Feb 16, 2011 9:03 PM ---
    Update: tracked down my issue... Message and CommandData have an auto int id! Oh, wow, I couldn't have done that more wrong, from a client perspective, if I'd tried :p

    Well, the good news is I don't think there's anything wrong with core Persistence- it actually did exactly what it was supposed to, given the criteria I gave it. It was constantly generating new CommandData and Message instances for me, including auto generating a unique id (how they were getting by the PK!)

    It was also adding them as contained objects, again, like I asked- masking this issue, because everything was actually working just fine.

    The take-away here, other than me having to break global data, is that there are probably a couple of uber-bloated "command" and "message" tables on your machines.

    I'm going to go ahead and make both of them contained (and make their string id the id, ditch the auto id), so those two tables will actually go away completely in the next release.

    Other than that, I may also break the lists (where the actual data that's being used is stored), since I'll be changing the id of the contained classes.

    All of this is motivating me to at least make some basic method to migrate or automatically reset global entities. I'll try to do this before the next release, so the data breakage is transparent to you all.

    Now that I know what's going on, I'm less terrified by what I've been seeing- the data is all safe (and that's always the most important thing to me!), but I can't imagine this is helping performance any.
    --- merged: Feb 16, 2011 9:05 PM ---
    Also, though I'm going to think about this more, this particular situation involved such a crazy edge-case combination of a client's use of various annotation knobs and dials, as well as a very particular and hard-to-catch error in how the API was being used, that I'm going to really have to give some thought as to how Persistence could keep someone from doing this. But, really, like I said- it did exactly what I asked it to do. :p

    And, furthermore, I could probably imagine a use-case where you'd actually want to set something like this up. This just wasn't that use case.
    --- merged: Feb 16, 2011 11:35 PM ---
    The more I look into this, the more I think I may need some new type of containment model for this use case entirely.

    Each PluginCommand is a node in a tree- is it needs to be able to reference other PluginCommands (one, it's parent).

    Each PluginData contains a list of PluginCommands- this data is stored in pluginCommands, one row per root command for each plugin.

    The problem is these contained rows reference unique PluginCommand entity instances- these get stored separately in the command table. These represent all of the sub-commands for all plugins, basically.

    And that's where things break- Persistence is currently smart enough to use both effective ids when pulling back references out of "command" to bind to their parents in "pluginCommands", however it's not smart enough to make a dual-key PK for that table, since it thinks it's just any other instance.

    This will cause problems with plugins using this system that use the same command name, even if that command is a sub-command of an otherwise unique command- a definite problem.

    I'm going to try to fix this in a way that extends basic Persistence functionality to cover this as a use case, but I'm not really sure what the fate of this command system is, at this point, in the long run- so I don't want to spend too much time on it, specifically.
    --- merged: Feb 17, 2011 1:25 AM ---
    Hrm- Nah.

    That would be a nice case to handle (a contained tree, basically)- but it's really very complex.

    So, I'm just going to reference the things with their auto int id's, which seems to have been my original solution to this problem- I just forgot to remove the "contained" tag from the annotations, basically.
    --- merged: Feb 17, 2011 2:16 PM ---
    Making some good progress here!

    I haven't been able to repro the infini-save issue, but I also haven't gotten into NetherGate debugging, specifically, yet.

    I found a couple more flaws in the global data for Message and CommandData. Let this be a lesson to you in the importance of getting your data right! You can always change it later, but, man- it's so much easier if you can get it right the first time :p

    So, basically- there is no need at all for PluginData to have Lists's of Messages and Commands - I only did this to keep them encapsulated by PluginData's id (effectively a way to give them a two-field id - the plugin id, as well as the message id).

    However, now that they're just full-fledged entities with an auto gen id, this really isn't necessary.

    So, now, when you get your PluginUtilities instance, it automatically populates a little plugin-specific cache of just your Messages and Commands, hashed by message id (not the autogen id) for easy lookup- since that id should be unique within the context of your plugin, this should all work out great.

    And seems to be, so far!

    There were also some bugs in how this complexity was actually being handled- more, even, then what I fixed and mentioned previously.

    Once thing that these bugs had caused, and which the logging has now made apparent, is maybe not "infinisave" but certainly "unneccesave". :)

    I can now start up Persistence, see it create all the global data- I shut things down, it saves it all (the object counts match). I start it back up, it loads it all (object counts match, again), and then-

    And this is the best part, since apparently it's never been this clean until now....

    Nothing happens.

    Since I didn't do anything, none of that data has changed- so this is perfect. Previously there were some bugs in the complex way I was managing the Plugin/Message/Command tree, and this was causing some of those objects to get unnecessarily marked dirty (or, in worse cases, duplicated- see post above).

    So, now this all seems to be fixed, which is great. I'm going to start looking into reproducing this NetherGate saving issue now, see if I can get that to happen- if it's a recurring issue, it'd be a huge performance win for that plugin, and that plugin is in dire need of some performance help!

    For anyone keeping track, I haven't really touched any "core" Persistence code in a bit. There's some data migration work in there, but the entry point is commented out because it's not really there yet.

    All of his stuff is a middle layer that I'm not sure how we'll handle if the core persistence engine gets into Bukkit core.

    This would include all the PluginUtilities stuff- so the Message handling, Permissions wrappers, Commmand/sub-command dispatch system, etc- basically, whatever they don't take from Persisistence is probably going to move on over to Gameplay.

    This also includes the actual "/persist" UI, "/su" functionality and "phelp" though, too- all the built-in commands, basically.

    and I'm not really sure where that should live, if they don't want to take it- doesn't belong in Gameplay. So, maybe there will always be a PersistencePlugin around for that, who knows?

    Anyway, just dropping in for a quick status update... things are going well.
     
  21. Offline

    ssechaud

    So can I not just uninstall and start from scratch rather than all this manual fixing?
     
  22. Offline

    NathanWolf

    PERMISSIONS PERMISSIONS PERMISSIONS

    Ok, now that I have your attention ... want to try out bukkit.permissions?

    Try this: Bukkit Permission / Persistence integration test

    You need to use the supplied CraftBukkit.jar.

    From an API perspective, there is no change- it's a seamless integration.

    From an admin perspective, start to learn bukkit permissions if you haven't already! Permission nodes are now encapsulated in player profiles, which is basically the most atomic unit that Persistence deals with.

    More on this later, I really just slapped it together and wanted to share.

    Not really tested so much- the user/group persistence stuff seems to work, I have not actually had time to set up a permissions config file... :p
     
  23. Offline

    QQCucumber

    I'm getting a very odd, and inconsistent, error message. Sometimes it happens, sometimes it doesn't.

    [WARNING] Missing a prefix or suffix for Admins

    or it will say Default

    Any idea why it does this?

    (Btw, I'm not a coder, I'm using it as part of Spells. I just figure it's Persistence since it happens at the exact second Persistence does things.)
     
  24. Offline

    NathanWolf

    Wow, yes- please remove all that quoting! Just delete the "Persistence" folder under plugins when you upgrade, that's all you need to do to start fresh.

    The upgrade instructions are for if you want to keep your data.
    --- merged: Feb 17, 2011 10:52 PM ---
    That's nothing that I'm doing- all my log statements are prefixed with the plugin they come from.
     
  25. Offline

    QQCucumber

    Look at it in full:

    2011-02-17 12:55:45 [INFO] Saving wand-commands.txt
    2011-02-17 12:55:45 [INFO] Persistence: loaded 4 objects from global.player
    ....
    2011-02-17 12:55:45 [INFO] Persistence: saved 1 objects to global.player
    2011-02-17 12:55:45 [WARNING] Missing a prefix or suffix for Admins

    Any idea?
     
  26. Offline

    NathanWolf

    No, still not me- please clean up your original post if you can.

    Those Persistence logs will go away if you update- they're harmless, but they're debug statements that I should not have left on.
    --- merged: Feb 17, 2011 11:09 PM ---
    More permissions stuff

    If you're interested, you can see how I'm integrating with bukkit.permissions here:

    (persistence/bukkit.permissions integration branch on github)

    Or, just look at the commit if you want to see the diffs, might be easier if you just want to see how to use bukkit.permissions.

    --- merged: Feb 17, 2011 11:10 PM ---
    And here's the ZIP again, in case it got lost in the chatter.
     
  27. Offline

    QQCucumber

    Okay, cleaned it up. I downloaded the jar file, but.. well, I'm a bit concerned. The one I have is 154kb, and the one I downloaded is 2771kb. Why did the file size go up so significantly?
     
  28. Offline

    NathanWolf

    EDIT: Uh... I missed a decimal point there, sorry. I thought I could account for that in sheer code size growth, but I cannot.

    Don't download the jar file directly, anyway- go update wherever you're getting Persistence from, if you can- NetherGate.zip, for instance, from the NetherGate thread.

    I'm pretty sure the one here should be up to date, and not 2.7MB... let me just go look into that real quick.... (I mean, you're right, I already checked ... let me figure out why now)

    Thanks for cleaning that up :)
    --- merged: Feb 18, 2011 1:29 AM ---
    Okay, Permissions Talk!

    So, now that I have a moment, I want to talk more about permissions, and where I'm at right now with it.

    From what I understand, the Bukkit team has been waiting for a good user/group management solution to come along. I (and I think a lot of people) didn't really have this full picture- permissions have been done for a while, but a way to effectively use those permissions, and a plugin API to interface to the permissions store, did not exist.

    This is my current understanding, anyway. I haven't really had a chance to get in-depth with the Bukkit team on this, see if they like my implementation, etc.

    If they decide they like it, and they feel comfortable with Persistence being the (or at least the first) user/group management system, then I imagine the permissions branch will get merged to master, and I can release this.

    At that point, I will drop support for Permissions.

    I will keep a link to 0.41, which will hopefully be the last version with Permissions support, in case anyone needs to maintain support for that.

    I will do my best to keep 0.41 API compatible with all current Persistence plugins for as long as possible- I don't anticipate this to be an issue. However, I will not actively maintain a version of Persistence with Permissions integration.

    So, eventually, you're going to need to get your admins to switch to bukkit.permissions, because you'll start to miss out on some crucial features like automatic data migration, and on-the-fly reload.

    Feel free to post any questions here- if we feel we need a separate thread for this, I can make one in the appropriate place.

    Also, if the Bukkit team likes what I've done here, but not how it's integrated with Persistence, I can easily pull the user/group system out into a separate plugin.

    However, that plugin would rely on Persistence anyway, so I'm not sure I care to do that- user/group management is a fairly well established concept, now that the hard work of permissions support has been done- I'm not sure I want to actively encourage the idea of multiple plugins for that. So, unless I'm convinced otherwise, I'd like to keep this stuff core Persistence functionality.

    Thanks for your attention!
    --- merged: Feb 18, 2011 1:38 AM ---
    Ok, some quick use instructions, and an addendum: I don't mean to imply that I want Persistence to "own" user/group management- not at all!

    Just to be a "hub" for it. PlayerGroup is a global DAO, like the rest of the Persistence-managed objects. You can get() one, get() a PlayerData, add a player to a group, get a ProfileData, use that profile to set up grant/deny permissions on a player or group, parent a group to another group, etc.

    Then, just put() your changes like normal- Persistence (any anyone else keep an eye on permissions stuff) will instantly get the update, since we're all sharing the same in-memory cached instances.

    So, this opens up a world of possibilities for user management plugins- I just don't think we need another user management framework, provided I can get this one right.

    Now, for the core Persistence functionality- there are some new commands I probably should have mentioned right away:

    Code:
    /group create <name> : Create a new player group
    /group add <player> <group> : Add a player to a group
    /deny group <name> <profile> : Set up deny access*
    /grant group <name> <profile> : Grant access
    /grant player <name> <profile> : Direct player perms
    /deny player <name> <profile> : Deny player perms
    A word on profiles and grant and deny:

    If you're not yet up to speed (and I have to admit that I'm not, entirely!) permissions are now encapsulated in a "player profile"- this is basically a dynamic data object that is used like a variant key/value data store/tree, in essence. It's very complicated, and not really the area Persistence manages, so go read the docs on that.

    Anyway, if I've done this right, the idea is that you use these profiles to grant/deny access to permission nodes- which can be scoped strings just like you're used to, as always it's up to the devs how they want to use pnodes.

    A profile that is used with "grant" behaves as you'd expect- any permissions applied to that profile will apple to that group or player.

    Using "deny", that profile will instead revoke permissions from a player, which takes precedence over any "grant" profiles they may have.

    This way, you can set up wildcard permissions for a player, but cherry-pick away individual rights if you want. It should allow for some really flexible arrangements.

    Ok, I have to get back to work- I'm hoping I can test all this out with CrowdControl tonight.
    --- merged: Feb 18, 2011 2:00 AM ---
    Ok- looks like I embedded sqlite in the jar by mistake (Permissions too... oops)... it seems to work fine, but it definitely bloats up the jar. I'll get that fixed up when I get a chance.

    Thanks for pointing that out, @QQCucumber! I have no idea how I didn't notice that, in sheer upload time alone- I'm not exactly on a speedy connection out here the boondocks where I live...
    --- merged: Feb 18, 2011 2:30 AM ---
    Also, I really should say that it's not "drop-in".

    Much like the new command system, I think there is YML work to be done- meaning that Persistence doesn't even support its own permissions system internally, yet.

    I'm going to play around with the branch more, see if I can get a YML file worked up for Persistence itself, then see if can make a profile YML and lock myself out of using persistence commands. When I get a chance.
     
  29. Offline

    QQCucumber

    I got it from Spells.zip in the Spells thread, I redownloaded it and it's the same file still.
     
  30. Offline

    NathanWolf

    Yeah- I haven't gotten a chance to update it. There is just an extra file in there- it shouldn't cause any issues, I've been (apparenly) using it myself that way for a while now.

    A JAR is very similar to a zip file- just an archive. I accidentally put the libs in the archive - probably have been doing this for a while, since I moved my dependencies into the workspace- I'll definitely get it fixed up ASAP.

    Meantime, don't let it worry you- as far as I know it's not a big deal, bukkit will load the jar and register it for classloading- but the extra jars within it (I believe) will be ignored.

    Anyway, as a quick update- now that I get in here and play with this user/group system, it's not really functional :)

    I've got permissions added to Persistences' config.yml for at least the basic commands, but once I got to testing I started finding all kinds of "I was in a hurry" type bugs.

    Fixing them up now, going to stop updating here in the meantime, I've been way too spammy. It doesn't seem like anyone's really jumping on this so I'm just going to take it easy for a bit :)
     
  31. Offline

    flerndip

    I'm running latest CB and Persistence started throwing this a few minutes ago, presumably when CB updated.


    2011-02-18 13:41:06 [INFO] Serious_Business [/149.166.9.138:60797] logged in with entity id 360
    2011-02-18 13:41:06 [WARNING] Persistence: Error updating table pluginCommands: [SQLITE_ERROR] SQL error or missing database (table pluginCommands has no column named commandId)
    2011-02-18 13:41:06 [INFO] INSERT OR REPLACE INTO "pluginCommands" ("pluginId", "commandId") VALUES (?, ?)
    2011-02-18 13:41:06 [WARNING] Persistence: Error updating table pluginCommands: [SQLITE_ERROR] SQL error or missing database (table pluginCommands has no column named commandId)
    2011-02-18 13:41:06 [INFO] INSERT OR REPLACE INTO "pluginCommands" ("pluginId", "commandId") VALUES (?, ?)
    2011-02-18 13:41:06 [WARNING] Missing a prefix or suffix for Default

    I'm running the lastest Persistence as of this post.
     

Share This Page