MCStats / Plugin Metrics (R7) - Easy & advanced Plugin Statistics

Discussion in 'Resources' started by Hidendra, May 25, 2012.

Thread Status:
Not open for further replies.
  1. Offline

    Hidendra

    It's still missing some information but:

    https://mcstats.org/api/1.0/LWC
    Shows basic info, starts in last 24 hours etc (it's still missing 30 minute servers and so on)
    https://mcstats.org/api/1.0/LWC/graph/Locale
    Will get graph data for the graph "Locale" (which happens to be a custom graph)

    About the never ending maintenance messages since last night, I was "attempting" to clear out 24.2 million rows from the graph table which aren't needed anymore (currently it has 61 miln rows). It didn't work out so well and MySQL crashed, which caused corruption and it appeared to not want to recover so easily. I've been transferring a backup now and am waiting for it to finish still, it shouldn't be much longer now ( I should have kept a copy of the initial backup I used so I didn't have to re-transfer it if I needed it but oh well, too late now ! )

    Everything is ok again, I'll hold off on deleting 24 million rows for another day :D

    As well, new custom graphs are now hidden by default -- if you want to make these viewable you will need to use the admin panel!

    EDIT by Moderator: merged posts, please use the edit button instead of double posting.
     
    Last edited by a moderator: Oct 29, 2015
    Jozeth likes this.
  2. Offline

    Hidendra

    Hello,

    This is a short post on graph generation and how it has been affected by growth.

    Over the past few months more and more servers have been connecting to MCStats / Plugin Metrics. As well, more and more plugins are using the service and it is becoming what I believe to be something that most plugins are seen using.

    The core of the service lies with graphs – and inherently – generating said graphs. A large amount of data has to be aggregated and normalized to be properly graphed and because of this how I was generating graphs was somewhat expensive and flawed.

    To make the problem more clear let’s tackle an example. If someone were asked to grab some X data for a set of plugins, it would probably look something like this:
    • loop over all plugins
      • run a query to count how much X has occurred
    It can even become more complex than that, for example what if you wanted to know how many servers came from a certain country (using an example in our usecases)? Your first attempt might look like this:
    • loop over all plugins
      • loop over all countries
        • run a query to count how many servers originate from this plugin AND country
    Now obviously, this is expensive. Very expensive. Say we have 1000 plugins and 200 countries (MCStats has 192), that is 200,000 queries before we even consider what we do with the count returned by this query.

    However, for a few plugins, this will most certainly not be a problem and you just ignore the issue that lies therein because it’s already really quick. You’ll never see it become a problem … right? ;)

    In the case of MCStats / Plugin Metrics, we’re getting a huge amount of plugins (as of right now, around 1560 in the database) and to use the simple route of loop over all plugins, run X does not cut it and slows down a lot as you add more plugins.

    As of September 20th graph generation was taking over 10 minutes – this is quite unacceptable and can cause locks and other messes you do not want to deal with. So what can we do?

    Case in point, take a simple query such as (simplified from the real query):
    Code:
    SELECT
        SUM(DataPoint) AS Sum
    FROM CustomData WHERE ColumnID = ? AND Updated >= ?
    
    ColumnID is an ID unique to a plugin for custom data, so if you have 2 plugins A B, A has 10 columns, B 20, this query will be ran 10+20 = 30 times to generate data. This can get very, very high as you add in lots of plugins with lots of custom data!
    Code:
    SELECT
        Plugin,
        ColumnID,
        SUM(DataPoint) AS Sum
    FROM CustomData WHERE Updated >= ?
    GROUP BY ColumnID
    Now what does this do? This eliminates the need to loop over your plugin set or even over anything, only the result set. This means that instead of executing potentially thousands of queries, you only use 1 query to fetch ALL data and it makes graph generation time entirely independant of how many plugins there are (and it doesn’t abuse the MySQL server)

    This optimization has been the sole cause in how graph generation could be reduced from over 10 minutes to less than 60 seconds without touching the database.
     
    Lolmewn, hawkfalcon and Comphenix like this.
  3. Offline

    nala3

    Is it just me or is mcstats down? It seems like it has been for a couple days now :/
     
  4. Offline

    Jozeth

    Same here...
     
  5. Offline

    Sefyre

    Hello!

    I apologize in advance for the length of this post (and the possibility that I might be posting in the wrong place and if so, please guide me to the correct place), but I need help in regards to understanding and fixing a [Metric]-tagged error (and possible related gameplay issues) that has suddenly started appearing on the server console and I haven't had luck in Googling for answers or trying to troubleshoot on my own.

    Very recently starting from September 21, my friend's Bukkit console suddenly started throwing up a particular [Metric]-tagged error that looks like this:

    Code:
    2012-09-22 15:09:27 [INFO] [Metrics] Software caused connection abort: recv failed
    and this error will pop up sometimes during server launch as the various plugins are being enabled like so:

    Code:
    2012-09-26 00:03:55 [INFO] Query running on x.xxx.xxx.xx:xxxxx
    2012-09-26 00:03:55 [INFO] [Metrics] Software caused connection abort: recv failed
    2012-09-26 00:04:17 [INFO] [ExperienceMod] [AutoUpdate] No new version detected.
    2012-09-26 00:04:17 [INFO] [ExperienceMod] [AutoUpdate] Version online: 2.5.0
    2012-09-26 00:13:22 [INFO] Friend[/x.xxx.xxx.xx:xxxx] logged in with entity id xxx at ([blah] insert location here)
    2012-09-26 00:13:56 [INFO] [Metrics] Software caused connection abort: recv failed
    2012-09-26 00:15:34 [INFO] Sefyre[/x.xxx.xxx.xx:xxxx] logged in with entity id xxxx at ([blah] insert location here)
    and otherwise throughout the duration of the game like so:

    Code:
    2012-09-26 02:23:55 [INFO] <Sefyre> we're in the general right area
    2012-09-26 02:23:55 [INFO] [Metrics] Software caused connection abort: recv failed
    2012-09-26 02:24:00 [INFO] <Friend> oh good
    In relation to it (at least it seems to be in relation to it, though it makes no sense to me), we've also been experiencing a strange series of gameplay issues that were never existent before these errors started popping up.

    1) Getting Stuck in a wall of mossy cobble
    2) Getting Stuck in a wall of soil blocks and receiving Suffocation Damage
    3) Walls of a Ruins and its Floor Suddenly Turning Transparent for no reason
    4) Worsening Player Connectivity Stability
    5) Player's Actions and Locations Not Being Updated Correctly

    Whether those gameplay troubles are simply a matter of coincidence or whatever else (server and/or gameplay lag, for sure, I could imagine giving such problems), I honestly have no idea, but I figured I might as well make mention of them, too, just in case.

    Trying to Google for answers for an entire afternoon has led us nowhere in terms of trying to troubleshoot this error as all of the other errors I could search for all had to do with Socket-related errors or other plugin-related errors or Minecraft-itself errors and I was wondering if you may be able to help us figure this particular error out and maybe answer a few questions if you have the time.

    IS this a Plugin Metrics-related error? If so, why is it happening? Exactly what kind of an error is it and how does it affect the game? Are any of the recently-spawned gameplay-related problems that I mentioned possibly related? Can my friend and I fix this and if so, how?

    Thanks in advance!
     
  6. From what i guess, that may have been where mcstats was down and experiencing problems. That i think may explain the recv failed errors. The other errors you are experiencing are not caused via Metrics, but instead another plugin, or possibly lag. Check with plugins that may alter a players position
     
    Sefyre likes this.
  7. Offline

    Sefyre

    Thanks a lot for the response! :)

    We temporarily disabled the Metrics in the plugin that we use that uses Metrics once I saw that MCStats was down for a while. Interestingly, the problems literally all went away for the most part and that was honestly the only thing we changed.

    We'll definitely investigate the other plugins (doing so as I type this), but for now, it seems like a good bit of the issues have gone away for some reason or another now that we simply disabled the Metrics for the time being.

    In lieu of this, it seems (from my end of observations) that when the plugin was having trouble contacting MCStats/sending data to MCStats/interacting with MCStats, it was also affecting the game as well.

    If that's the case, then do you think there might ever be a functionality for Metrics where if there is trouble with MCStats then the Metrics will automatically stop trying to contact MCStats for an X amount of time or until it is re-enabled by the server admin?
     
  8. Offline

    Lolmewn

    Is it gonna be up soon?
     
    Butkicker12 likes this.
  9. It could be the plugin itself, the way it starts mcstats, it might report custom data to mcstats which interferes with it, its all dependent on the plugin.
     
    Sefyre likes this.
  10. Offline

    Sefyre

    Thanks again for the response! :) Do you think, then, that I ought to report this issue (even though it's pretty much stopped now) to the plugin(s) that uses MCStats/Plugin Metrics?
     
  11. Offline

    Icyene

    Hello!

    I've been using MCStats for a while now, and have just recently come across a problem. Whenever my server loads, all plugins using metrics log a "Error: connection reset", or something among those lines. This occurs both on a Windows XP server and a Ubuntu Precise Pangolia server. Additionally, the MCStats page does not load in any browser I have tried (Firefox, Chrome, Safari, IE). At one point it managed to load the Essentials page, but all the graphs are empty, with "No Data" in all the text fields. I have tried resetting my cache and, when that failed. switched computers. Same problem.

    Lastly, my plugin was having ~100 servers running, then in the span of half an hour it dropped to just under 10 (shown by the signature image, which seems to still work). Surely this can be a coincidence, but I find it unlikely as its been averaging ~100 for the past few weeks. A few hours ago it has resumed to ~50.

    Is this just me, or have others been experiencing this problem?

    Thanks in advance!
     
  12. Offline

    Jozeth

    Same here, nothing to worry about.
     
    Icyene likes this.
  13. Offline

    devilquak

    I've been unable to access the Metrics site for a few weeks now, just wondering, is this just me, or is everyone else having the same problem?

    If it's not just me, I hope I'm not sounding whiny when I say I miss it and want it back... :(
     
  14. Offline

    silvinci

    It's impossible to register new plug-ins. It doesn't accept any plugin name.
     
  15. You have to have metrics on the plugin already, and up on a server so it can get data from the plugin, then you register it. Some of my plugins took about a day of data before i could add it
     
  16. Offline

    silvinci

    AH! Thanks, mate. ;) This should really be noted done somewhere...
     
  17. Offline

    Jozeth

    He can only add admin rights to plugins on his database... Simple really.

    And it is: https://github.com/Hidendra/Plugin-Metrics/wiki/Admin-Portal

    "This also allows you to request plugins to be added to your account once you add it to the MCStats backend."

    EDIT by Moderator: merged posts, please use the edit button instead of double posting.
     
    Last edited by a moderator: Oct 29, 2015
  18. Offline

    silvinci

    Nope, not really. Why can't he add the plug-in to his database, when I want to register it? :D
    Adding something to the MCStats backend for me sounds like registering it to the MCStats website. I'm used to register my stuff first, before I can use it. Gathering admin rights later on isn't a common practice on the interwebs.

    The notice on the add-plugin page should be something like: "Before you can register your plugin, it must have sent some data to our servers. Refer to the wiki for integrating MCStats in your plugin."
     
  19. Offline

    Jozeth

    It's on automatic, once data is sent from a server, it's put onto the database (https://github.com/Hidendra/mcstats.org/blob/master/sql/database.sql). He'd have to make 1 row in 4 tables for you. When you could just add it to your plugin...
     
  20. Offline

    silvinci

    You didn't get, what I wanted to say. On every other website, I first have to register my plugin/API/programm, before I can send data to their service, like Dropbox, Facebook, Twitter, etc.
    But on MCStats it's the other way around, which confused me. The note on the add-plugins page didn't clear things up, because for me a "backend" is the actual website.
    I proposed a pull request, which puts the note a bit more expressive.

    And reffering to the automatic stuff: I don't think that Twitter and co manually add the APIs either.
     
  21. Offline

    Jozeth

    I get what you mean, but the Admin feature is optional. So you can add an author and edit the graphs if necessary.
     
  22. Offline

    silvinci

    I don't think that the admin feature should be optional, but we're on different opinions here, which is ok. :D
     
    Jozeth likes this.
  23. Offline

    TobyZ28

  24. Offline

    Jozeth

    Same here.
     
  25. Offline

    KeybordPiano459

    Yeah me too. Everything got deleted before about 6am this morning EST, and my plugin stats haven't updated since 7pm.
     
  26. Offline

    Jozeth

    It's just broken...
     
  27. Offline

    ToastHelmi

    How long does It take to add a plugin to your account (usually)
     
  28. Offline

    Muffin89

    Amazing website service, great work i must admit :)
     
  29. Offline

    KeybordPiano459

    Meh I usually wait a few hours...
     
  30. Offline

    ToastHelmi

    now my plugins are added
    i think i only had to wait so long because of the failure
     
Thread Status:
Not open for further replies.

Share This Page