[Tutorial] How to create a Network Framework

​

Welcome again to another one of my posts, this time we will be covering one of the most looked at aspects of Bukkit which i have not found a tutorial for, "How to create a Network Framework". The fetaures this tutorial will cover are:

- Maven (Done)
- Github / BitBucket (Started)
- Databases. (Done)
- Permission Systems. (To Be Done)
- Gem Systems. (To be Done)
- Utilities. (To be Done)

Lets start.


Using maven is probably one of the best choices you can make as the developer of a network as it allows you to create a repository which can be hosted on your website easily as Bukkit does so your partners can use the Jar files located in that repository such as the Main Framework of a Network which is probably extended in every one of the plugins the network uses unless its not custom coded.

Creating a maven repository can be hard but once you do it its the most recomforting thing ever, if you dont know how to create a Maven repository check out this link http://www.mkyong.com/tutorials/maven-tutorials/. After setting up the maven repository you are almost done with that as simply you have to code the Whole Framework and do the following commands on the command line or command shell:

Code:

cd {where your project pom.xml is located}
mvn clean install
mvn deploy

That will compile your framework and deploy it to your website so your fellow developers can use the same jar. But beware creating a repository as it may be hard, if you wish you can PM me or contact me on skype (DevRosemberg) so i can create one for you or even host one for you.

If you wish to connect to a special repository and have access to deploy to it you will need to add something like this:

Code:

<settings>
<servers>
  <server>
  <id>yourname-ftp-repo</id>
  <username>mailhere</username>
  <password>passwordhere</password>
  </server>
</servers>
</settings>

Into your settings.xml located in your .m2 Folder which should have been created in your user Folder. In my case:

Code:

C:\Users\Dev\.m2

Then in your pom.xml you will add something like this:

In your DistrubutionManagement part:

Code:

<distributionManagement>
        <repository>
            <id>yourname-ftp-repo</id>
            <url>yourftpurl</url>
        </repository>
    </distributionManagement>

Note: Where it says yourname-ftp-repo is exatly the same name as in settings.xml

And in your Build part:

Code:

<build>
<extensions>
            <extension>
                <groupId>org.apache.maven.wagon</groupId>
                <artifactId>wagon-ftp</artifactId>
                <version>2.2</version>
            </extension>
        </extensions>
    </build>

Also using Maven like this will allow you to have stored lots of Jar files in a server without having trouble finding old versions of a plugin, as you will see in the next picture where it is showing versions of TheCosmosCore that were developed (note it is an old repository and the new one is way larger in the old repository there were around 25 Jar files whereas in the new one its 10 times that, only 7 fitted in the screenshot).



I think that would cover the whole maven section so lets continue.


In my opinion using either of these services (https://github.com/ or https://bitbucket.org/) Is a great choice as they both support the creation of the so Called 'Development Teams' to be able to share code within the network or publicly as the Overcast Network does it. I myself prefer to use BitBucket because you can create Teams free and because of the graphic display which is really user friendly, for example in BitBucket how Cosmos is organized:



Note the Icons which GitHub dosent support, of course both of them are really good but we just use BitBucket for its GUI and as its free and easy to use. But also, GitHub has a really nice IssueTracking system so we also are using GitHub to do the Tracking. (As you will see in the next picture the Issues is private as its not finshed yet and the rest of the projects are quite old, we are not using GitHub anymore to store our code).



For the lazy ones IntelliJ and Eclipse both have plugins & integrated Git & Mercurial support so it will be really easy to switch from GitHub to BitBucket or the other way around.

//To Be Continued


Databases, one of the most tricky parts of creating a network as you have to consider Safety, Speed, If it will be able to contain all data and more.

A really good choice for Databases can be MongoDB (http://www.mongodb.org) Because its a really fast Database System which has Java Drivers, is really easy to install in Ubuntu or CentOS and is extremely safe and never fails. Some examples of networks using this are:

- Hypixel Network
- Arsenal Network (In-Dev)
- Overcast Network

Those networks chose MongoDB for the reasons i already told you. Maybe learning the mongo programming style can be a bit hard and confusing but learning this: http://www.mkyong.com/tutorials/java-json-tutorials/ and http://www.mkyong.com/tutorials/java-mongodb-tutorials/ will prepare you perfectly for using Mongo. In the network i develop we use Mongo because its easy, fast, safe and can store a LOT of stuff in it. If you need help using MongoDB you can check the Mongo website itself or contact me on skype (DevRosemberg).

In the following example you can see a simple way of creating a DatabaseManager, if you use it i would appreciate credit:

Code:java

package com.devro.Currency.Utils;
 
import com.mongodb.DB;
import com.mongodb.DBCollection;
import com.mongodb.MongoClient;
 
/**
* Programmed by: DevRo_ (Erik Rosemberg)
* Creation Date: 03, 01, 2014
* Programmed for the Currency+ project.
*/
public class DatabaseManager {
private static MongoClient mongoClient;
private static DB db;
private static DBCollection players;
 
public static void connect(String host, int port) {
try {
mongoClient = new MongoClient(host, port);
} catch (Exception e) {
System.out.println("Couldn't connect to database!.");
}
db = getMongoClient().getDB("Framework");
players = getMongoDB().getCollection("Players");
}
 
public static void createNewData() {
if (mongoClient != null) {
mongoClient.close();
}
mongoClient = null;
db = null;
players = null;
}
 
 
public static MongoClient getMongoClient() {
return (mongoClient);
}
 
public static DB getMongoDB() {
return (db);
}
 
public static DBCollection getPlayers() {
return (players);
}
 
}

So then in our onEnable() method we can do something like the following:

Code:java

DatabaseManager.connect(getConfig().getString("Host"), getConfig().getInt("Port"));

And then start working on what we want like for example a Gem System which we will talk about later in this tutorial.

Although MongoDB can be a really good database many people and network prefer to use the Legendary MySQL (http://www.mysql.com/). Using MySQL might not be as hard to learn Mongo though it can be tricky and confusing, for our luck -_Husky_- made a great tutorial on how to use MySQL for your plugins (http://forums.bukkit.org/threads/tutorial-using-mysql-in-your-plugins.132309/) where you can get to learn a good amount of MySQL and he provides code for it. MySQL is also really easy to install like Mongo on Ubuntu and CentOS and also has Java Drivers. Here i leave an example of a MySQL.java file created by Husky, all credit goes to him:

Code:java

package code.husky;
 
import java.sql.Connection;
 
import org.bukkit.plugin.Plugin;
 
/**
* Abstract Database class, serves as a base for any connection method (MySQL, SQLite, etc.)
*
* @author -_Husky_-
* @author tips48
*/
public abstract class Database {
 
/**
* Plugin instance, use for plugin.getDataFolder() and plugin.getLogger()
*/
protected Plugin plugin;
 
/**
* Creates a new Database
*
* @param plugin
* Plugin instance
*/
protected Database(Plugin plugin) {
this.plugin = plugin;
}
 
/**
* Opens a connection with the database
*
* @return Connection opened
*/
public abstract Connection openConnection();
 
/**
* Checks if a connection is open with the database
*
* @return true if a connection is open
*/
public abstract boolean checkConnection();
 
/**
* Gets the connection with the database
*
* @return Connection with the database, null if none
*/
public abstract Connection getConnection();
 
/**
* Closes the connection with the database
*/
public abstract void closeConnection();
}

Code:java

package code.husky.mysql;
 
import java.sql.Connection;
import java.sql.DriverManager;
import java.sql.ResultSet;
import java.sql.SQLException;
import java.sql.Statement;
import java.util.logging.Level;
 
import org.bukkit.plugin.Plugin;
 
import code.husky.Database;
 
/**
* Connects to and uses a MySQL database
*
* @author -_Husky_-
* @author tips48
*/
public class MySQL extends Database {
private final String user;
private final String database;
private final String password;
private final String port;
private final String hostname;
 
private Connection connection;
 
/**
* Creates a new MySQL instance
*
* @param plugin
* Plugin instance
* @param hostname
* Name of the host
* @param port
* Port number
* @param database
* Database name
* @param username
* Username
* @param password
* Password
*/
public MySQL(Plugin plugin, String hostname, String port, String database, String username, String password) {
super(plugin);
this.hostname = hostname;
this.port = port;
this.database = database;
this.user = username;
this.password = password;
this.connection = null;
}
 
@Override
public Connection openConnection() {
try {
Class.forName("com.mysql.jdbc.Driver");
connection = DriverManager.getConnection("jdbc:mysql://" + this.hostname + ":" + this.port + "/" + this.database, this.user, this.password);
} catch (SQLException e) {
plugin.getLogger().log(Level.SEVERE, "Could not connect to MySQL server! because: " + e.getMessage());
} catch (ClassNotFoundException e) {
plugin.getLogger().log(Level.SEVERE, "JDBC Driver not found!");
}
return connection;
}
 
@Override
public boolean checkConnection() {
return connection != null;
}
 
@Override
public Connection getConnection() {
return connection;
}
 
@Override
public void closeConnection() {
if (connection != null) {
try {
connection.close();
} catch (SQLException e) {
plugin.getLogger().log(Level.SEVERE, "Error closing the MySQL Connection!");
e.printStackTrace();
}
}
}
 
public ResultSet querySQL(String query) {
Connection c = null;
 
if (checkConnection()) {
c = getConnection();
} else {
c = openConnection();
}
 
Statement s = null;
 
try {
s = c.createStatement();
} catch (SQLException e1) {
e1.printStackTrace();
}
 
ResultSet ret = null;
 
try {
ret = s.executeQuery(query);
} catch (SQLException e) {
e.printStackTrace();
}
 
closeConnection();
 
return ret;
}
 
public void updateSQL(String update) {
 
Connection c = null;
 
if (checkConnection()) {
c = getConnection();
} else {
c = openConnection();
}
 
Statement s = null;
 
try {
s = c.createStatement();
s.executeUpdate(update);
} catch (SQLException e1) {
e1.printStackTrace();
}
 
closeConnection();
 
}
 
}

With that code we can now do something like:

Code:java

MySQL MySQL = new MySQL(plugin, "host.name", "port", "database", "user", "pass");
Connection c = null;
 

And in our onEnable() method add the following to actually open a Connection:

Code:java

c = MySQL.openConnection();

With that we can now work on Gem Systems and More because it will be easy, i would extremely recommend that you take a look at Husky's Post as it is full of information and documentation about MySQL.
//To be Continued

 

Seems it will be a great tutorial!
I'm waiting for the rest.

 

ArthurMaker I wont keep you waiting more as at night im going to work on it a lot and each time i can ill go adding stuff, if you want to keep in contact or something Follow me (I have no followers )

 

DevRosemberg
Your amazing.

 

BungeeTheCookie Thanks if its intended in a good way and if not i dont get it?

Updated Main Post, added an example of a Database Manager.

EDIT by Moderator: merged posts, please use the edit button instead of double posting.

 

Reasons for choosing MongoDB:

I can't even think of one other database that doesn't also meet this criteria, for varying definitions of 'fast'.

Same. Every database is equally easy to install under ubuntu, centos, or any other decent linux disto, they all have packages in the package managers.

http://hackingdistributed.com/2013/01/29/mongo-ft/
TL;DR MongoDB is garbage.

This "tutorial" is really lacking. The only real useful information is in the maven tutorials it links to. I had to get to the section on databases before I even understood what you meant by 'Network Framework'.

I really don't see why a maven repository is necessary at all, though I suppose it could be useful if you're developing a lot of interdependent plugins with a group of developers. still hardly unique, or especially relevant to a server network. The same can be said about a database, all plugins can benefit equally from a database, not just server networks.

You need to start by defining what a server network IS and what it's core components are. what makes it different from a single server, why would you want to do it, and THEN how can you go about doing that. none of those questions are addressed by this tutorial. its just a link to a maven tutorial and a brief mongodb example.

 

metalhedd First of all, It is incomplete and i still need to finish it. I wills tart providing code and defining everything about a network and AFAIK you dont know much about MongoDB as Hypixel, Nexus, now minecade is switching to it, the cosmos and Affinity use it.

 

Overcast Network also use MongoDB ^^
That they say on Twitter and their Forum.

~ Marcel

 

Overcast Network also run's on MongoDB

 

DevRosemberg The amount of people using something doesnt't make it good. A lot of other networks probably use other databases and by your reasoning these have then to be considered equally as good. I can only agree with metalhedd and read some other highly rated articles claiming that mongodb is not always a good choice and there are a lot of issues with it.
As far as the tutorial aspect of this goes, you should always provide alternatives for the user to play with so he can always form his own opinion.

 

kumpelblase2 I know and im already planning to write about MySQL as it is also a good database.
RainoBoy97 True, will add that.

 

As far as I can see (Out of the last like 4~ days), You've made a bunch of really useful tutorials. You seem to see into my mind and make a tutorial explaining exactly what I want. Can't wait for the To be done's to be done. Thanks.

 

Caprei Haha Thanks a lot Caprei, im really flattered.

 

Cool addition to the forums. Nice tutorial.

 

Hypixel and Nexus may be popular minecraft servers, but in the big picture of the entire internet, they're very small potatoes, and not particularly worthy examples of technology done right. That's not to say they're bad by any means, but they're no amazon, google, facebook, etc. They don't need to scale to millions of users. They don't need to have 99.99999% uptime. They don't need to worry about losing a few database records (it's only a game, not sensitive customer data or credit card transactions). Because of that, they're free to use basically whatever technology they want/can afford with very little risk.

Now, back to mongoDB... it really is probably the very worst of the bunch when it comes to 'safety'. did you read the article I linked?

That's pretty bad. Almost any other database would be 'safer.'
Anyway, I'm not trying to criticize your efforts at all. You've obviously put time into these tutorials, but I can only see them doing more harm than good. If anyone successfully gets through this, they'll have all the infrastructure, and no idea what to do with it, or what any of it actually does, why they need it, or what they've just accomplished.
You don't just 'create' a network of servers with all of time invested into infrastructure (a Maven repo?! Really?!) before you have any need for it. I would venture to guess 99% of the people who attempt to follow this tutorial will never, ever need their own maven repository, but it's at the top of the list, just begging to waste several days of their time in frustration trying to figure out how it all works (maven is a hideous beast, seriously), and in the end, it will have very little practical use. Anyway, as I was saying, you don't just up and say "I'm going to learn how to create a minecraft server network today" the very idea is absurd, and this tutorial kind of encourages people to just do it anyway, without even a prerequisite like: You should probably already have a server with a thriving community, and a talented team of sysadmins to actually operate all the infrastructure that you're about to blindly install.
Also, how can you operate a server network without a proxy like lilypad or bungeecord? there's no mention of those in the planned topics. All of the networks you've cited as examples use one or the other, and it's one of the only things that actually is a required part of the stack, everything else you've listed is unnecessary.. I can only feel pity for the people who are considering following this tutorial hoping to end up with something like the next HyPixel

 

metalhedd
1) You wrote Hypixel incorreclty
2) There are people who like this tutorial, if you dont i really think you should erase those posts you did.
3) If you have any problem with this tutorial simply dont post in it.
4) If you have any problem with MongoDB You yourself describe it because for the same reasons i can go and say facebook is something bad because they store peoples data.

 

Sorry?

I won't be erasing any posts. My warnings are more useful than your tutorial.

don't post in a public forum if you're not prepared for criticism.

I have no idea what you're trying to say here. But I do agree that facebook is something bad... so, I guess we agree on something?

 

Nvm

You are the only one who thinks like that

I am prepared for Criticism but as we see kumpleblase is the only one that supports you as people enjoy reading it.

Ok?

 

Hey, I enjoyed reading it too, but that doesn't make it a good tutorial... The other people who have posted in this thread obviously just aren't aware that the advice is actually bad. kumpelblase2 and I are the only people who have expressed opinions who actually have more than 1 plugin under our belts, and probably some experience actually running servers.

 

DevRosemberg I don't see, why he should erase them as he clearly stays on the topic and just proposes his own opinion about it.
I do feel kind of the same way as creating a "network framework" (I wouldn't even call it like this) is really more than what you've talked about and even what you've planned. In fact, the prerequisites metalhedd talked about should be mentioned in giant words because otherwise this whole thing would make no sense at all.
Moreover, I'm think that people who actually are at a point of planning such a network and have those prerequisites are more than aware of how they will do it, even without a tutorial. Honestly, when you have come that far, it's not really that big of a deal to do those things yourself (sure, it takes time and effort, but the will have an idea on how things need to go). In fact, I needed most of the things you proposed here without having a network. Hell, I needed a database for my first plugin - and most of the servers will probably feel the same.

To the topic of mongodb...

Show Spoiler Hide Spoiler

, I personally have a big issue with not having transactions like probably any sql database. Because this can screw you over when storing large amounts of data, which you probably will when you are planning to build a network. Imagine your server dies in the middle of a query, mongodb will be like: "ohhh, I misses these entries? I'm sorry, I can't do anything about that". This is what I personally dislike the most and should be not be considered 'safe' by all means. And they can't make it safe, because they'll lose speed (which is probably the biggest plus for mongodb) and thus won't be the fastest any more compared to other nosql or even sql database.

TL;DR : There really has to be an alternative proposed here.

Show Spoiler Hide Spoiler

One last thing: Think about who will really read this and use this in all honesty. Yep, not really the people who are in a position where they can make something out of this, but instead beginners or less experienced programmers (I don't see a reason why a high-skilled programmer should come here: the help you get is underwhelming because of how few good programmers are around). What will happen? They will do this and at the end will sit there and be like "what now?". They don't have the knowledge to lead or continue on such a project. It just doesn't make any sense to me to open up such a topic here, because the audience is not suited for it.

and which are not plugins that have been made 50 times already.

Because like I said, the targeted audience does not have the proper knowledge about those things. Who is allowed then to criticize the content of a tutorial when you only accept people who don't have this knowledge or experience?

 

+1 to kumpelblase2 and metalhedd here. OP should be careful posting stuff.

 

metalhedd I see where you're coming from with the "MongoDB isn't safe" thing, but as you said before, it's not horrible if someone loses 1 point or achievement or something as much as it would be for 1 person's order to be removed from accompany like Amazon. Also, the whole issue you described can easily be fixed by changing the WriteConcern on Mongo, and you can also completely disable the journaling feature to make sure all data is immediately written to disk.

 

metalhedd Like macguy8 said, There is no such "Imposible to crack" Database as all of them have been hacked. Kind of quoting what mac said, it dosent quite matter for someone to lose 1 point or earn some plays, it can be fixed and big companies also have this errors for example ive lost some Orders on Amazon and actually i have lost full access to an account in eBay for a bug in their code.

 

None of this changes the fact that you recommended a system that's inferior to the more traditional database, AND completely ill-suited to the task at hand. for amazon it may be a tradeoff between scalability and eventual consistency, but a minecraft server network, no matter the size, has no need for such scale, and thus no need for mongoDB. People that do use it can make the concious decision based on their own needs. If and when they make that decision, they can also do the research about things like WriteConcern which macguy8 mentioned (which, btw make mongo slow... negating the main advantage you gain from it).

DevRosemberg 'hacking' and/or 'cracking' have absolutely nothing to do with this conversation. We're talking about the integrity of your data in the event of a power loss, hardware failure, network outage, etc. things that happen *ALL THE TIME* which mongo is really bad at recovering from, but which most traditional databases have no issues with. While these traditional database may be a little bit more difficult to scale to millions of concurrent users, no minecraft network will ever need that capacity... so you're sacrificing integrity for scalability that you'll never use. it's just a bad choice, plain and simple. Just like it's a bad choice to setup a maven repository before you even know what it does, or to write a tutorial about server networks without ever mentioning how to actually connect 2 servers together.

 

metalhedd I feel like this is more the advanced end of setting up the server network. I'm sure I can find hundreds of tutorials about BungeeCord and connecting two servers together.

 

because that's the only thing necessary for a server network. nothing in this tutorial has anything to do with the subject, and certainly nothing to do with 'advanced server network management'

 

metalhedd As much as DB speed matters, you don't see every network using Redis do you? For a minecraft server DB queries don't need to be fast, they need to work 99.999% of the time, which they do. As well for WriteConcerns, you should be doing DBqueries async anyway so there'd be no reason for it to matter.

 

DevRosemberg
I don't see how this is really a "framework" containing 2 classes. Frameworks consist of interfaces, classes, objects and singletons to provide access easily to a utility. For example, if CaptainBern had a singleton class (which he doesn't because he prefers to be fancy and use his framework directly) then it would be a fine example of a fully featured framework, consisting of the basic wrappers for reflection.

Wait actually, he technically does have a accessor class, called BukkitTool (I remembered )

All in all, confusing title, un-indented code, unneccessary parts such as setting up GitHub and Maven - Leave that for you blog, not on a Bukkit oriented forum. And finally, inaccurate or misleading information: saying that MongoDB is
"Really safe and never fails" will probably throw people off. It did for me until I did my own background research. MySQL is often an industrial database, not MongoDB which is, put in a nutshell, "simpler".

I'd also like to point out that this is a database tutorial that has nothing to do with actual networking, in the sense of SeverSockets and HTTP.

Sorry for being the technical guy, but I wouldn't say this is the best written tutorial out there.

 

I personally prefer MongoDB over MySQL.

Is there really need to criticize a guy just trying to help the community by creating a nice guide? We all have personal preferences.

 

"advanced end of setting up [...]"? I don't agree with that at all. The things mentioned in here are basic level, really. Databases: everyone should've come across them at some point before doing such a network, otherwise I don't know where you've been the whole time. Same for github and maven, these are parts which should already exist before even thinking about making a network. The whole tutorial would assume that you build this from the ground up with no current server running at all, and for me it feels senseless to make a network without having a server running already.
I tell you what I would expect from an "advanced end of setting": How would the infrastructure look like; How can you easly manage all of them or even something simple like "Do I even need to make a network instead of having one big server".

Generally, the target audience is, in my opinion, chosen wrong and since the tutorial is made for that audience, it is not really good. Sure, it's not done yet, but maven, github and a database being main points, I don't see how this can actually cover it to a good extend. I certainly don't know everything that is needed for a network, but I can say that the things proposed here certainly are not part of it, because they're part of the foundation to how to make running a SINGLE or even small server easier. Maybe even a step further, making the development of serveral chained programms (if needed: in a group) way easier and this wouldn't even have the need for a server at all. Thus this is not needed for the topic at all and should be a prerequisite instead. Everything what comes after that would probably be better suited here.
Continuing, who, who is in the position of making such a network, still needs code examples for such basic things? I certainly think no one, only those who shouldn't think about making a network, because they have not enough experience to handle something like that yet, and thus making it completely unnecessary.