And ... we're back.

Hey guys -- just want to update you all to let you know that we successfully switched over to the new servers. While RPI players already noticed that we'd moved, we decided to wait a few days to make sure everything worked properly before we officially announced. Over the past few days, we've re-added the dynamic map (when our servers couldn't handle the load we had to replace the map with an image), 6X-ed the number of turns for RPI a day, and un-paused Rice. The result? We're functioning fine and considering re-launching the ILC soon.

A lot's changed though. From a non-technical perspective (which is how this article is written), it probably sounds easy to add additional servers. But, there's a good deal more that goes into scaling out than just throwing more money at hardware. We had to rework some of the back-end -- particularly the application's interactions with the database -- so as to allow it to operate on many computers approximately as it would on a single one.

So how is this done? I usually save the gory details for the second date, but we arranged our servers into what's known as a master/slave configuration. We separated our servers into master and slave servers. The slaves hold read only copies of the data on the masters and constantly check with the masters for updates. Meanwhile, the masters write new data. So, any time you sign up for a new account, move a unit, or post a chat, your action is written to the master. When you load a page, chances are you are loading content served from a slave. If you're curious for a longer explanation, by all means Google (you should probably leave safe search on if you're in class).

Certain challenges must be overcome during this process. Unless appropriate precautions are taken, a system might fail if a user hits one server, and then another upon reload. Data can show up incorrectly if a slave database hasn't yet copied changes from the master. A data collision can occur if the database tries to insert two pieces of data to the same point. Or, a major bottleneck or single point of activity can bog down everything as traffic increases. GoCrossCampus was originally built to operate on a single server - and so we spent a lot of time planning solutions, researching and coding to make the transition as smooth as possible.

Over the past month, we optimized, cached and moved towards stateless interactions with the database. We added indexes to our tables, made code more efficient, fixed a lot of small bugs, separated out static content (CSS, JavaScript, Images) and began transmitting compressed content.

We've had to handle a couple minor glitches - and had to take the site down for a few minutes once. But we haven't run into any major problems yet. Overall we've been very pleased. Since December 1st (the day we switched to the new servers), visits have more than doubled, page views have 5X-ed and our servers are holding up strong.

On that note, let the games (re-)begin!

Almost...

Wanted to give everyone another update on getting some games back up. We're well into the testing phase now, and it should not be long at all before we can get RPI's game over to the new system. Once that happens, we'll want to make sure that everything is working as expected. Then, we can get back to playing and growing as planned. :D

We're committed to having everything work well once it goes back up, which is why we're taking a while with it. This is also why we might seem a little less focused on immediate problems than we were three weeks ago. Since we're switching systems soon, it doesn't make too much sense to spend our time on short-term fixes rather than making the long-term solution better.

If you need further explanation, just check out today's XKCD.

An Update

So, it's been almost two weeks now since we started putting games on pause. I think you're due an update.

First, it's probably a bad idea to unpause anything before Thanksgiving break is over. People want to think about getting the hell away from college. They want to think about sleeping twelve hours per day, eating turkey with the family and spending their Christmas shopping cash.

They don't want to receive a half dozen commander contact emails and worry about whether they placed their troops on the side of the library.

Plus, this just gives us more time to make the lag not suck. Every day we go before we unpause, we can spend our time optimizing, testing and fixing instead of figuring out why turn processing suddenly removed all commanders from their posts and replaced them with MIT students.

Also, a heads up: when the games restart, we are launching the first inaugural GoCrossVideoContest! It's open to all schools, but check out the RPI game page for more details:

GoCrossVideoContest details on RPI Game Page - Click "Learn More!"

Ruminations upon a Pause

"Game Paused."

A gamer on his lunch break. Writing a paper. Be back later. Out.

Or as AIM put it, "away from my computer."

Pausing has connotations, and I'm not entirely sure what they mean. I'm a bit afraid they are gamer-ish. Not only gamer-ish, but evocative of '90s single player games. Like the guy who can't tear himself away for long enough to save and return later.

But I mean, what else should we say? "Time-out?" No. Teams call time-outs. The people who run the game don't call time out.

"Game Paused."

The verbiage here is, of necessity, imprecise and awkward. We're not sure what we're doing, but we're pretty sure it's something new. We're not an online game. It may say so on our home page right now, but the last month has shown us that our identity lies closer to a pickup game of coed football than World of Warcraft.

Okay, fair.

But what else do we say, then?

Game postponed? Too ominous. Baseball is postponed due to rain. They often don't reschedule the match.

Game on hold? What does this mean, exactly? It jumps out there, grabs you, and says "We don't know what we're doing with this thing." Too indecisive.

Game Break? "Now let's go to Chris Berman in the studio for an update on RPI!"

Game respite? Okay, now I'm just using thesaurus.com.

Now, let's get nerdy. The capitalization is critical here. "Game paused." means something totally different from "Game Paused." "Game paused." implies that we (the admins) took decisive action to pause the game, i.e., it is read as "[The] Game [was] paused [by us]." "Game Paused," on the other hand, is somewhat of a cultural meme developed within gaming circles and spread throughout society during the mid to late 90s. It's not a blatant example of such, but it does resonate as a state rather than a description of an action. That is, it is read instead as "[The] Game [is] Paused." rather than the more active "Game paused."

The implication is critical here. While "Game Paused." achieves the immediate communication objective better than "Game paused.", it nonetheless associates us with a specific gamer mindset and culture.



Note: To answer the inevitable cries that we spend more time thinking about this crap than actually fixing things, I assure you that many people with far more software development skills than me are extremely busy right now.

Hi, Rice!

As if our server hamsters needed any more work, GXC is now inviting Rice University to the party. Rice, like Yale and Harvard, uses the Oxford Residential College System. As the birthplace of GXC, our game flourishes among residential colleges. The reasons for this aren't rocket science.

1. Residential colleges are the universal center of social life at these universities. Unlike other schools where students are split between the Greek scene, campus groups, and local groups, every student at Yale and Rice consider themselves a part of their college.

2. Residential colleges tend to be small, equally sized groups. It's a lot easier to recruit among a group of 100 when facing another group of 100 than it is to recruit among a group of 3,000 when facing another group of 30,000. Or another group of 300. Or another group of 3,000.

3. Residential colleges concentrate teammates in a geographical area. This emphasizes the "locally social" aspect of the game and allows physical recruitment (at dining halls, going door-to-door) rather than only electronic.

That said, the game seems to work at a variety of places -- whether we're looking at schools without Res Colleges (e.g., RPI) or between schools themselves (e.g., the Ivy League Championship). It's just a bit tricker to find the right way to divide the teams so that every student is fighting for something they identify with. Ultimately, this is something that should be done entirely by students without any interference from us.

But first, I have to go feed some hamsters.

Saboteurs! Be gone!

The GXC grapevine recently dropped us an intense bit of news -- apparently certain players, or "script kiddies," with support from the team, are engaging in the act of bringing our servers down with malicious code...all in an attempt to to prevent other teams from having a fair chance at playing.

Now that's just dirty. Remember my post about fighting with honor and playing fair? If you're not gonna listen to the Samurai, at least listen to Steven Colbert.

"Don't be a GXC saboteur. America will hate you."

Have some decency people. There are a lot of players that genuinely enjoy playing our game and are upset by your acts.

Needless to say, if we catch any players (or teams) in the act of trying to bring our site down -- or even condoning it, we will disqualify you.

GXC is a sport as much as a game. Play by the rules.

Our Server Woes

Hey everybody -As you probably know, the engines of war have been grinding to a halt around 7 pm each night. As you probably know, this sucks. We’re chalking this up to the overwhelming popularity of both our games thus far, as over 40% of on-campus students at RPI and 10% of all Ivy undergrads are actively playing. This is awesome, but we unfortunately don’t have Facebook’s server farm.

For now, we suggest not waiting until last minute to place your armies or issue your orders. It could save you the possible disappointment of not getting your orders in before turn processing starts.

The situation is quite sad actually. Did you know that if you give a MySQL database too much work, it just refuses to continue! We get emails from it saying:

"MySQL server has gone away."

...on vacation, perhaps? We don't know. But we always pray that it come back quickly.

Just so everybody knows, we're working really hard to optimize our code. For example, the Flash map currently makes 800 MySQL queries on each page load [update: we've since gotten the number down to 18 queries, but are trying to make it even smaller]! Think about that happening for 5,000 visitors, all hitting refresh simultaneously, while our server is already struggling to find some RAM to start turn processing properly.

We've made some strides in optimization and cutting down the number of SQL queries our game page makes significantly (although we haven't published the changes yet). We've already started the process of researching how we want to incorporate more servers into our system architecture. Hopefully in the near future, all of you will see a noticeable improvement in site performance at peak times.

Needless to say, of course, this is a better problem to have than having no one visiting the site at all. So we thank everybody for being dedicated players and coming back consistently to participate!

And we really appreciate how understanding all of you have been during this whole experience. Thanks for bearing with us!

Very gratefully,
The GXC Development Team