AggroChat #361 – Huge Queues Tiny Servers

Featuring: Belghast, Grace, Kodra, Tamrielo, and Thalen
Tonight we are back after taking last week off for Tam’s wedding.  We also record an exceptionally long show because we got into the topic…  and were not watching the clock terribly closely.  So it clocks in at around an hour and forty minutes of talking about Amazon’s New World release and all of the good and bad that comes with that.  While this makes for an interesting show…  it makes for some pretty boring show notes.

Topics Discussed

  • New World Launch
  • Infrastructure Woes
  • Great Game Though
  • Server Transfers
The post AggroChat #361 – Huge Queues Tiny Servers appeared first on Tales of the Aggronaut.

New World Launch Post Mortem

Good Morning Friends. I find myself still here and still playing New World. I did however take a break and play some Final Fantasy XIV… while the New World client was up and idling in the background. Yesterday I logged in over lunch to about a thirty-minute queue, and that was enough anxiety to have me just leave the client open all day long. After yesterday’s blog post, I took to “the twitters” and threaded up some commentary for anyone who might not read my blog. My advice this morning is still pretty much the same as that thread, if you are not already bought into the game then I would highly suggest just waiting to pick it up until the server congestion situation is more reasonable.
The New World team took to their Twitter account and posted this message, trying to put a positive spin on the current situation. The tweet talks about how many servers they created on the fly and how they are working to increase capacity on each server. It also goes on to say that if people just pick any available server now they are working on a way to get folks transferred onto a server with their friends. The problem here is this fundamentally is not how an MMORPG works. You need those friends the most in that early rush as you are attempting to do dungeons and collaborate on getting folks geared. Even if you are not actively playing with other folks, there is something necessary about just knowing that friendly voices are out there waiting on the other end of guild chat. There is something deeply flawed in the design of this game and I will go into detail a bit but first, we need to talk through some numbers. Right now at this very moment, we have the following server capacity:
  • United States East Region – 158 Servers available
  • South America East Region – 35 Servers available
  • Europe Central Region – 195 Servers available
  • Asia/Pacific Southeast Region – 40 Servers available
  • United States West Region – 49 Servers available
Looking through this you notice some strange data points. First off the EU region has significantly more servers than I would have expected at this point based on the sizing data from other MMORPGs. The United States West Coast region seems grossly undersized to what I would normally have expected based on my experience with other games as well. In grand total, you have 477 servers and if my math is correct a maximum capacity of 954,000 players could be playing the game at any time. There is a maximum single server capacity of 2000 players based on the metrics we have available to us, and there are many realms that have more players in the queue than actively playing on the server.
In yet another tweet thread, I surmise that there is something about their design model that is fundamentally easier to scale horizontally than to scale vertically. This 2000 player limit seems to be important, as does the 100 players in a single company. That tells me this game was more or less designed for 20 companies to be duking it out for the resources of Aeternum, which makes me wonder if New World more or less started its design cycle as a slightly larger scale Battle Royale game, and morphed into an MMORPG game as players violently rejected its PVPcentric design. We know without a shadow of a doubt that AWS can scale upwards extremely well, and you can very well keep throwing resources at a problem by adding more processing power, memory, and disk space ad infinitum. So that tells me there is something on a software level that makes it a challenge to scale above 2000 players in a single server.
Let’s talk about that 2000 player number a bit. Once upon a time both Everquest and Dark Age of Camelot showed the current player population of each server on the login screen. Quite frankly this was something they were both proud of because you have to understand… coming from games with a maximum player population of 16 in a single match, having thousands of players was quite the feat. The average peak for those servers generally speaking was in the 2000-3000 player range. As we move forward to the modern era of MMORPGs, I have heard that 6000 players are the point at which the login queue starts to kick in on a World of Warcraft server. Final Fantasy XIV requires a bit more factoring to come up with some sort of estimate. Based on the player knowledge I have… most people do not use the Steam client, and I would guesstimate that at most 1/5rd of the player base connects that way because you have a huge group that plays through their console. There are 68 total servers in FFXIV, and Steam recently reached a peak concurrency with the game of about 67,000 so for sake of making things simple let’s call that 1000 steam players per server which again is going to be a deeply flawed number. If we are taking as an assumption that the number represents a fifth of the player base then my guess would be that would give you around 5000 concurrent players per FFXIV server.
So you might be asking yourself… Bel, why did you go through all of that contorted math that you know is going to be completely wrong? Just trying to paint you a picture of what the modern MMO landscape looks like as far as player population caps, and how 2000 per server seems extremely small based on the demand already existing in mature games. In an immature game, there is going to be even more of a burst of players that needs to be dealt with as folks are currently hungry for any new games coming out regardless of genre. New World also is a special game, because it represents the first completely new “Western” MMORPG in years. Western players have developed a pretty sour taste in their mouth for the microtransaction hellscape that are Korean and Chinese localizations. Based on the recent flurry of activity in Final Fantasy XIV, there should have been signs that the community was primed and ready to latch onto something new and fresh given that the entire World of Warcraft player base seems freshly unmoored.
So once again I fall back on, that there is something specific to their design model that indicates 2000 players as a specific unreachable boundary. Otherwise knowing what I know about AWS, it would have been far simpler to add additional processing power to the servers that were already provisioned than to keep spinning up new nodes. There has to be some reason on the software side that they are not doing this, however, and it makes me question if they are confident that they can scale upwards. Alpha and Beta testing didn’t really give them an idea of what a real-world player demand looks like. We went into those tests just happy to be playing, and not much caring if we were playing with our core group of friends. However, an MMORPG launched is a totally different beast, and players organize into dense clusters as everyone tries to make landfall on the beach that their friends already arrived on. Having more servers doesn’t really do much to dilute the demand placed on those first few servers.
Now we are going to swing back around to why the servers per region numbers look fucked. Amazon shot themselves in the foot with this launch and rolled the game out in a manner as to encourage players to dogpile a limited set of servers. Amazon rolled the servers out by region in a staggered manner… causing the EU and NA East servers to fill up completely because once players are established somewhere, there is entirely too much inertia to get them to move elsewhere when more appropriate servers are opened. The schedule looked a little something like this:
  • European Servers Opened at 11 pm PDT on 9/27
  • South American Servers Opened at 4 am PDT on 9/28
  • Asia/Pacific Servers Opened at 4 am PDT on 9/28
  • North America East Servers Opened at 5 am PDT on 9/28
  • North America West Servers Opened at 8 am PDT on 9/28
The first servers to come online were in the European Data Center, and there was a mad rush to get in and reserve your name. This is another flaw that added to the problems we are dealing with, but the fact that character names are globally locked and not tied to a specific server. As result, there were a lot of folks that popped into those servers to create a character reservation, and likely a handful of people who just wanted to play the game period regardless of server environment stuck around and started playing the game. The next big avalanche came when the East Coast servers opened, and since there is a relatively insignificant difference for most players between the two environments… every single major guild chose to start the game at this point.
Regardless of how painful the launch might have been for some players, the staggered launch and the fact that there was a land run on virtual real estate… aka player and company names made everything that much more important that you got there first. The problems that we are dealing with in this New World launch are absolutely problems of Amazon’s making, in part due to sheer lack of experience in dealing with the MMORPG player base. There is only about a 100 ms ping difference between me playing on my NA East server and playing on a European server, which means for most people the choice of where they landed was largely meaningless. New World was treated like a highly localized experience when time and time again the players have proven that they are a global audience.
So here we are with this botched launch and questionable game design… that we could easily walk away from were it not for the fact that when you ARE able to get into a server the game is damned addicting. New World is a great gameplay experience when it is operating under optimal conditions. My hope is that they have engineers working on how best to scale the servers, and as one of my guild members commented… if they doubled the capacity per node most of the queues would go away instantly. I think they need to do precisely that and at the same time offer free server moves at a company level. Let entire companies transplant themselves on greener pastures, because quite frankly unless they specifically are holding territory there isn’t a lot connecting people to a specific node at this point. I hope they can make this work because the game legitimately is good. We haven’t had an MMORPG launch that was this solid in a very long time. My advice stills stand for anyone waiting and watching from the wings. Keep waiting. If they manage to pull out of this tailspin I will be the first to raise notice that it is time to start paying attention again. Until then, however, all that waits for you is a whole lot of frustration and anxiety buffered only by just how good the game actually is. The post New World Launch Post Mortem appeared first on Tales of the Aggronaut.

Bigger is Better

Good morning folks! So I thought I would start this morning’s post with a little bit of a tip. In New World, there are a lot of resources nodes that come in various shapes and sizes. This is not just a cosmetic thing, the larger the node the more resources you will get from it. This means that if you ever roll up on a cluster of nodes, always focus on clearing them from largest to smallest. What is often going to happen is another player will roll up and start to harvest a node beside you, which means you are in a race to gobble up as many resources as you can as efficiently as you can. This for example is Iron and it is among the most hotly contested materials and given that your mining skill, your strength, your faction standing cards, and the quality of your tools all determine how fast you can gather as compared to other players. Always start on the big node and work your way down from there.
Yesterday I spent most of my time wandering around and mopping up lower-level quests that I probably should have already taken care of before now. Of course, while going on a walkabout… I would discover resource nodes that would distract me from my goal. This is the core gameplay loop for me and honestly what I enjoy about this game and games like Skyrim or the modern Fallout series. I set forth on an adventure with a goal in mind, but the process of traversing the distance in between provides a real meaningful journey. The truth is in these sorts of games I tend to use fast travel or mounts very sparingly because it is that cross country trek that really provides me with joy. In New World, I like that I pretty much still need resources that I can find rarely out in the fields and forests on the way to my destination, so stumbling across a stand of hemp for example means I am always going to stop and gather it before continuing onwards.
So far surprisingly I have only really encountered a single bugged area, and that is surrounding a quest where you need to kill a named mob down in a cave. The problem with this quest is that it seems like folks are camping the area constantly and the spawn time so infrequent that whenever we do get a pop… it is almost immediately killed by splash damage. I am uncertain how much interaction is needed in order to earn credit for a quest, but it seems like I am always short because I am pretty sure I am interacting and dealing damage but never actually finishing the quest. I am hoping to roll over there really quickly after publishing this post and maybe just maybe the early morning crowd has moved on past this area.
At this point, I have made it to level 23 and I am thinking about maybe taking a bit of a break from the game and returning to my leveling over in Final Fantasy XIV. The key turn-off right now is knowing that I pretty much have to idle with the game running in the background all day if I have any hope of actually getting some playtime that evening. Then what makes that worse… is knowing that it is highly unlikely that any of the people that are starting the game are going to be able to roll on our server and join us. This whole only 2000 characters per server thing is a pretty debilitating problem, and it threatens to kill the experience for a lot of players out there that will likely bounce, or spend two hours in queue and be unable to return the game. I think someday this will be a great experience, but until they sort out the technical limitations it feels like a really fun game that is only available for a limited few players who can manage to land on the servers in a reasonable manner.
As far as the server goes, the other company still holds First Light but is at risk of losing it. There is a war that has been declared because Marauders have taken the fort and make war on the territory to claim it. Based on what I just raid on Twitter it sounds like the Marauders are playing dirty as the Trading Co. has just had a number of their highest level members given temporary bans. I guess this is the problem with contested territory and games that more or less have automated enforcement systems. I have no clue which group is gunning for the territory because the fort doesn’t say which force claimed it, but I am going to guess it is the Spartan Legion that controls Windsward, and that they are going to be taking their progress in that territory and beginning to take over the map. Honestly, I would be completely fine with that group taking more territory because they seem to be charging fair tax rates. As a citizen of Aeternum and not a combatant… I have a completely different view of these territories because quite honestly all I care about is having a beneficial overlord that isn’t going to charge odious taxes.
I am greatly enjoying the game, but the fact that I cannot reliably play with friends… because my friends cannot reliably get into the game is somewhat damaging the experience. As I said earlier, I am getting tired of feeling like I need to stay connected in the background in order to save the said connection for the moments I can play the game. There is simply no viable way to make it through the queues during prime time hours and I think for the moment I might just move on to doing other things with my time and hoping that at some point the server sizing problems are fixed and we can all move forward together. As much hype is surrounding this game can easily turn bitter if these growing pains are not resolved quickly. More servers are not the answer… bigger servers that allow more people to play together are the only path forward. If Amazon cannot provide that, then this game will die on the vine. Otherwise, folks are probably just better off playing something like Valheim if they can’t actually experience this reliably with friends who didn’t take off work to log in at non-primetime hours. The post Bigger is Better appeared first on Tales of the Aggronaut.

The Forest is Burning

Well, folks, we colonized the New World yesterday and… I have to say on some level it is exactly what I would expect a land run on new territory would look like. For those who managed to log in, I think the experience was pretty solid. There was EXTREME resource scarcity for those who were there at 8 am when the servers turned on, and while everyone was trying to crit path their way to choosing a faction. I know largely because I was there with them because a last-minute change of heart meant I too was trying to get things set up and running and get a company functional for invites. Mind you I had no intention of doing this thing, but as we got closer to the launch of the game I started feeling that maybe that big company wasn’t really my scene.
All in all, though it was probably a good thing I had that last-minute change of heart. While we did not get as many invites as I wanted… it wasn’t for lack of desire. There were a lot of folks who just simply gave up on the server queue, and since I logged out around 9:30 pm CDT I also wasn’t really around for invites. I’ve tried to pass out officers to a bunch of people, but far as I can tell there isn’t really a way for people to see who is on in the company if you are not already in the company? One of my big takeaways the social structure of this game needs some tweaking. For example, I don’t have a Message of the Day set because if I start typing… the second I hit space my character jumps and it exits out of the interface. It also doesn’t support cut and pastes… so I can’t even “jury rig” it into working that way either.
Other than just some general concerns I had about the attitude of the leader, my main concern and what ultimately prompted my action is knowing that we were dealing with a very limited pool of possible invites. From what I understand the larger company filled up without actually managing to get everyone in that had signed up on the roster. I knew in my case that I had friends who were wanting to play this game that might not be able to create a character until the weekend, or god forbid a week late… and there would be nothing I could do about it. I refuse to adopt a “got mine, fuck you” attitude towards my friends. Back in World of Warcraft, I had a top tier raid guild courting me for one of their main tank slots… but I ultimately gave up on that notion and helped found a “friend and family raid” instead because it felt wrong to go off and abandon the people who helped me get where I was. I didn’t want to lead a company… but doing so absolutely made me feel better about the whole situation.
This is what our territory map looks like on Minda, the server I am playing on… and it is nowhere near as Syndicate dominated as I would have feared. Right now the Marauders aka Team Green controls the territory I hang out in Windsward as well as Monarch’s Bluffs. Over night the big company that I was going to join managed to snag First Light, which was the last of the four starter zones to be claimed. I am guessing they went for the route of raising 100,000 gold in order to buy the claim. Everfall is likely going to be in a constant state of flux because I believe when I was last there yesterday it was held by Covenant, and now is held by a Syndicate company with a pending War. I am impressed that Cutlass Keyes has been taken, but not shocked given that zone has a lot of higher-end resources in it and is ultimately in heavy contention. The rest of the zones are probably not held simply because folks are too low level to reasonably venture there.
As for me, I rushed ahead to start the company started and then idled in the background since I knew there was no way in hell of getting back into the game. It took me about 30 minutes in the morning and then I spent several hours last night leveling and managed to hit 18. There is something to be said about familiarity and knowing more or less where various resources are. I’ve had so many starts in this game in all four of the territories, but I think I lucked out and Windsward is probably my favorite. As a result, though I pretty much knew the critical path to get what I needed, and for example by the time I hit the first town I had already collected my 40 stone, 40 wood, and 40 leather for the later quests.
My real focus however has been getting my trades up so I could start making my own gear. Given that they changed things up quite a bit I am now focusing on Engineering because I really want to be able to make my own crafting tools. Ultimately I want to be able to craft Azoth Extraction tools in order to keep a steady flow of that for crafting or teleports. At some point, I will probably focus on Weaponsmithing and Armorsmithing, but with everything being so freaking cheap on the market board I can pretty much buy whatever I want without denting my bank account. My hope is by the time I do get up there in skills… the glut of cheap items will have ended and I can actually start making goods for profit.
All in all, for a launch day it went pretty smoothly. The only larger concern I have going forward is that I am not sure Amazon fully understands the importance of being able to play with your friends. Right now it appears that the servers are capped at around 2000 players, combined with Companies, in general, being capped at 100 players… makes me question if they understood properly the scale of a large MMOPRG or if we misunderstood what sort of game they were going for. 2000 players and 100 per player unit… is more like a larger Rust or Ark than a proper MMORPG and I wonder if that is ultimately what they thought they were building here? My hope is that the 2000 player thing is an artificial metric to help force players to spread out a bit, and that given time they will increase that limit. However for pretty much all of the servers last night, the queues were as large if not larger than the population of the server based on what I could see from this third-party tracking website. My hope is that they do some shifting and buffing of resources and that we can all actually get on the server we want to be on and play together. Ultimately that is the most important part of the MMO experience, is being able to play with a group of people that you enjoy playing with. I am glad I started Greysky Expedition, and I am happy that we are going to have room for folks… pending they can actually get on the server. So this was not a bad launch but there are some structural concerns I have going forward that would keep me from calling it a perfect launch. Did you manage to get in? What server are you playing on and what was your experience? Drop me a note below. The post The Forest is Burning appeared first on Tales of the Aggronaut.