[WEB/ADMIN/API] BukGet: The Unofficial BukkitDev JSON API

That's for md_5 to answer.

 

Agreed.

 

Something like when I want to run static analysis on all the plugins, which at the moment I have to:

1. Get your entire list of slugs
2. Get your page on each slug
3. Scrape the DBO download page for each project using the link in part 2, since you don't provide a download link.

The other core reason I have is if I wanted to write a search engine and needed low latency + complex queries, or do mass analytics on the description or some other aspect of every plugin. For this a full dump is absolutely crucial.

 

I finally got a workable version of the API up. Here are some progress screen shots:

 

Ok, I'm going to start contacting Devs that I know are using the API to start testing against the development build thats up now. I expect there to be some issues, however I hope everything by and large works w/o breakage. The only things that are significantly different are:

* Commands and Permissions are normalized out
* GenInfo Ids are now referencing the MongoDB document, and are not integers
* GenInfo actually shows changes now.

I'll be posting the URLs up on the OP

 

API Docs are now posted on the website.

 

Major code shift. the API and the parser are now separate modules. Furthermore, BukGet now has an organization in Github \o/

 

Can't find that address, might be handy if you can give us an external URL?

 

Lmao

 

Perhaps http://bukget.org/pages/docs/index.html

 

that's it

SteveMcGrath
{"description": ["PlayerIP is a plugin with commands..."], "plugin_name": "PlayerIP", "slug": "minecraftplayerip"},

on http://dev.bukget.org/3/plugins/bukkit/

Description is array. Is this a bug in the parser? I suppose it is, because I don't know why description would be an array.
If it isn't a bug, I can alter my parser, but otherwise I thought this might be good for you to know


Edit:
Except for the bug mentioned above, updating the GUI went flawlessly

Wrong version data for : dev.bukkit.org/server-mods/abitofrealism
hmmm, http://dev.bukkit.org/server-mods/activator/ too.
Seems like the version parsing doesn't fully work yet.

Is dev.bukget relyable to use already (except for the version parsing), or is there a possibility to release more stable beta builds to api.bukget.org/3 ?

Edit 2:
Small suggestion:
I'm using
http://dev.bukget.org/3/plugins?fie...ption,versions.version,versions.game_versions
But is there a way to only show the last version and game version? Now it shows all versions, which results in quite a big page.

 

Ill look into adjusting that. Generally speaking, the reason for this is bad plugin.yml, however I can fix it in the parser to just return the string value of the array. For example, here is that plugin.yml:

I just checked into this, the issue is the plugin dev. They have never updated their versions in their Jar files. I will contact them about the issue and hopefully they won't be soo lazy next time.

BukGet is parsing the version in the plugin.yml over the version on the site now. There are a lot of reasons to do this, however mainly for reliability. So this is not a bug in the parser, just lazy devs who dont update their plugin.yml files properly. I'm continuously amazed by the amount of badly coded plugin.ymls that exist.

dev.bukget.org will be moved into production in mid Jan. as I have been coding fixes for the new API code, I jave been automatically updating it. As the codebase is a ground-up rebuild, I can't easily graft on API3 to current stable without replacing api2 and api1. All of the APIs have been coded into the new codebase however.

Not from the plugin listing no. However you could always just accept only the first item in the versions array for each. The Dev API is now keeping stats on the plugins, so I don't think we would impliment this in a single call, mainly because the stats updating is hooked into the plugin details calls.

So I wrote a quick little script to help me manually fix these errors. If you see any more inconsistencies, then let me know.

Also Double-check the API, those should report properly now.

Fixed the link. sorry about that :-p

EDIT by Moderator: merged posts, please use the edit button instead of double posting.

 

The API result for the plugin list now works without problems
http://dev.bukkit.org/server-mods/accident-tnt/
Also displayed wrong.
However, the ones that are wrong are consistent in the version field. Always 0.1. Maybe you could use that to run an extra check on it? If you search the filenames a second time using regex for

Code:

\d{1,5}(|.\d{1,5}){1,4}

, or similar, wouldn't it work?
This regex would match x to x.x.x.x, each x between 1 and 5 digits. You might need to add allowance for some letters (b, #, beta). But you could use it if the first check/parsing didn't work.

http://regexpal.com/

 

you would assume it to be that easy..however here is the regex I was using: https://github.com/BukGet/bukget/blob/master/bukgen/bukgen/bukkit.py#L30

The problem is that a LOT of people don't conform to using proper versioning. this led to a lot of issues with the parsing. One of the many reasons why it was switched to parsing the plugin.yml instead. Either way, there will be issues that require cleanup. I also need to hope that the BukkitDev staff are checking for things like these.

 

Ok, well Bertware, I can't ignore this. There are some 484 plugins that are in the database like this, which is unacceptable. I'll have to figure something out, probably add in that logic. I might as well have a dbo_version that uses the regex and is copied to version if this condition is detected.

Ok, so I ran a quick little script to see how many plugins have 3 or more versions and only 1 version number...155 plugins came back. Here is the pastebin of the json and the code. The problem I can see with performing a simple check is that even if we patch the issue now, if plugin devs don't realize whats going on, it'll get worse later. BukkitDev staff really need to check to see if the plugin version noted on the upload matches the plugin.yml version IMHO.

JSON Output: http://pastebin.com/0LKKFxGj
Code: http://pastebin.com/n4G8eqLJ

EDIT1:

After working on it for a few hours, I have some code I'm testing in the repository right now that solves this issue I think satisfactorily. I'll update this thread when I have a better idea.

EDIT2:

Code now support this favoring of DBO_Versions under specific conditions. Everything appears to be working.

EDIT by Moderator: merged posts, please use the edit button instead of double posting.

 

I appear to have encountered an issue here, while trying to access this:
http://dev.bukget.org/3/plugins/bukkit/mcmmo?fields=slug,plugin_name,versions.download
i get an error: Error 500 Internal Server Error.
but if I omit the ?fields addition, then http://dev.bukget.org/3/plugins/bukkit/mcmmo works just fine
it seems to be the case in api1 and api2 as well, the fields just won't work.

 

I fixed the issue. It had something to do with the stats monitors that I added. Everything works as intended now

 

Seems like most of them are fixed now

 

Yayz! I'm running a full generation as we speak to pick up any stragglers. It should complete sometime this evening.

 

thanks! Here i was thinking i couldnt read instructions correctly
I think many people underestimate just how useful and easy to use this API actually is

 

Thanks for the kind words. The API has been through a few iterations to get everything right and has taken a lot of Man-hours of coding and work, and it's always nice to hear that people find it useful.

 

Just bumping this for everyone.

 

We all greatly appreaciate what you do. I'm sorry, I should have said this in a much more clear way. What you did is amazing
I wish I could help you out better then how I am currently, but I promise that I will do so as soon as I've got the means to!

 

I have to agree with Antariano about the awesomeness and amazingness of the API, without that there wouldn't be plugin managers that can show this much details etc (versions, names, authors, ...) nicely formatted

 

Appreciate it man, You guys were one of the first larger projects to utilize the API and helped to promote it to the usage it's at now.

Appreciate it!

To be honest, I'm continually amazed at how popular BukGet got in 2012 and am glad people are finding it useful. Now I just need to find a server to play on and just relax a little

 

I hope that with SpaceCP we'll be able to use it much more efficiently than we are currently Keep it up!

 

Well im happy to report that after about a week of usage, the new codebase has proven to be rediculously stable. There are still a few things that need to be cleaned up (mostly on the updater) however everything is running well, no memory leaks, and the incoming requests are all being handled well within tolerances (actually better than the old code!)

I do have some things on my plate to start working on:

* JSONP support for API3 (Yes, callbacks. Been asked for this a couple of times)
* Ability to delete items from the database that have been deleted in BukkitDev
* Support for multiple services (python venv + better config file handling)
* Code cleanup and dopcumenting (common.py is badly documented, among others...)
* GET MORE PYTHON DEVS ONBOARD!!!!!
* Get a JavaScript person onboard to write a simple GUI into the API.
* More stuff that I can't think of at the moment.

 

Popularity is starting to get phased into the API. More details when we have enough metrics to fill out day/week/month.

 

how can i search with the api3 a plugin with the "main"name? like "com.earth2me.essentials.Essentials" for essentials?
some plugins have the same slug-name and you doenst find the right on

like: this plugin the slug is "Stats" but the api-url is http://api.bukget.org/3/plugins/bukkit/lolmewnstats
when i'm looking for this plugin i get this site http://api.bukget.org/3/plugins/bukkit/stats

 

It's not something thats currently parsed, however could be added. Generally speaking however the only thing that I have found to be truly unique is the slug.

ADDED

Here is the commit for main to start being tracked. It will organically load up on newly changed plugins until friday, when I kick off a full index to get the updated field into all of the plugins.

https://github.com/BukGet/bukget/commit/5b48594c1c19e49c53a3e2d5b9a46a43d8967450