How indie developer Bit Dragon tackled cross-platform play in Hyper Jam

Hello, my name is Geordie Hall and I’m a co-founder of Melbourne-based studio Bit Dragon. I’m excited to be able to say that our first title, Hyper Jam — a fast-paced synthwave arena brawler with a dynamic perk drafting system — is launching today on PC, PS4, and Xbox One!

Although we’re only a small team, I’m super proud that we’ve been able to ship with local versus, private online, and cross-platform matchmaking (with extra local players). This game has been a truly awesome learning experience, and we’ve had to push ourselves in almost every way. I'm really glad we chose to go with Unreal Engine for Hyper Jam, as it provided our small team of three programmers the tools we needed to make this ambitious multiplayer game possible.

I told you it was fast-paced!

The road to cross-play

Like a lot of indie multiplayer games, our initial online experiments focused primarily on private lobbies using peer-to-peer listen servers. In our pursuit of better netcode we eventually decided to test out dedicated servers to see how they compared. After that initial playtest, we were all very impressed with how good they felt - particularly on Australian internet. Once we committed to a cross-platform launch, the expense of dedicated servers started to seem more justifiable, and the possibility of cross-play matchmaking convinced us it was worth taking on.

Cross-platform matchmaking itself has some major benefits, the most obvious being more players in your matchmaking pool. For a large game, this should decrease queue times and improve the player experience, but for a small game it could be the difference between someone finding a match or leaving in disappointment. Sharing a matchmaking pool can also hedge against a single platform becoming a ghost town if it doesn’t perform as well as others, which might temper bad reviews and increase the longevity of your game. If you manage to support cross-platform parties as well, then you’ve really broadened your audience!

So how achievable is crossplay as an indie? If it’s your first time tackling online services then there’ll definitely be a steep learning curve, and you’ll inevitably have to jump through more hoops than you were expecting. But the good news is that with so many companies working hard to consolidate and democratize online game services, it’ll only get easier over time! Epic themselves even offer a suite of cross-platform online services rolling out over the next year, adding to the list of options available to developers.

If you’re like us and have an online game on multiple platforms, that’d be a great fit for crossplay, read on to find out what we learned along the way.

Dedicated servers

A major prerequisite for cross-play is dedicated servers. If you’re already using listen servers peer-to-peer, then you might have noticed you can’t simply host a listen server on one platform and connect to it from another. Each platform generally has its own net driver, which handles NAT traversal on that platform (e.g., the SteamNetDriver uses Steam Sockets). But for cross-play, you need each platform to connect to a single server and “speak” the same language (e.g., using the standard IpNetDriver). If you still want to use listen servers for private matches (like we do), then you’ll need to change which net driver you use at runtime.

Although P2P works pretty well for private matches where players know each other, it has some inherent downsides that make it a poor choice for matchmaking:

• You’re reliant on the host’s internet connection for a good player experience — even with good QoS checks
• Host advantage, something we’re quite mindful of as a client-predicted fighting game
• Host migration is hard, and without it a host can easily ruin the game for others
• You can’t trust the host, which rules out ranked modes

Dedicated servers solve all of these issues, with the obvious downside that they cost money to host. This ongoing cost can be difficult to justify as an indie, but after committing to a cross-platform launch, we decided to make the investment.

If you’ve only used listen servers before, then dedicated servers might seem scary, but they’re actually quite straightforward. You can build a server for both Windows and Linux, but since Linux instances tend to be cheaper to run than Windows instances, it’s probably worth spending the time to get a Linux build working. Thankfully, UE4 makes this very easy! Once you install the cross-compile toolchain you can then include Linux as a target platform. Clang has historically been a bit pickier than VC++, though they’re slowly becoming more similar. You’ll also need to check that any plugins you’re using support Linux, and be disciplined about keeping gameplay logic out of cosmetic objects (since dedicated servers don’t run widgets, sounds, etc.).

By default, a dedicated server will run headless and simply log to a file, though you can add the “-log” argument to have it launch a log window as well, which can be very useful during development. Dedicated servers are also very easy to integrate into your Continuous Integration (CI) pipeline — just one extra UAT argument! When you need to test a Linux server, you can spin up a local VM or EC2 instance, though a Windows server should suffice for most development testing. You can even use dedicated servers in PIE, which is great for debugging Blueprints.

Cloud hosting

Once you have your dedicated server binaries and your netcode is running great, you’ll need somewhere to host the server so that clients can connect to it! There are lots of hosting options, including Amazon GameLift, Google Cloud Compute/Kubernetes, PlayFab, and more. The major things to look out for here are cost, scaling, and support.

You need to be able to scale your servers up and down to meet player demand, ideally scaling as low as possible during off-peak. Most of these systems will spin up a virtual machine that get assigned a specific IP, and runs N server processes, each of which listen on a different port. The server manager can then assign game sessions to each server process, keeping track of which ones are busy. Once the number of free server processes on an instance drops below a certain threshold, the system will spin up another VM, which will add another N available game sessions to the fleet. The scaling system should also assign new sessions intelligently to allow scaling down and make good use of resources (e.g., you don’t want five game sessions spread across five machines).

Most providers have some kind of free tier you can use to play around and try out different configurations, which is a great way to dip your toe in the water.

As an indie, cost is always a concern, and you’ll need to balance how much extra work you’re willing to put in to lower server costs. When looking at scaling costs, the main equation will be “players per server x servers per instance x number of concurrent players.” The GameLift pricing page has an example of a typical player demand curve, and shows how that affects the number of instances required throughout the day, although these numbers may seem high for an indie game.

Optimizing your game for both CPU and memory will increase the number of servers you can fit on each instance, lowering your costs. Keep in mind you may also be charged for network traffic, so optimizing your net code may actually improve your server bill as well as your gameplay!

You’ll want at least one instance running in each region to take on new players, but depending on your availability requirements, you might be able to take advantage of Spot/Preemptible Instances for additional scaling, which are instances that cloud providers can terminate at short notice in exchange for big discounts.

It’s also a good idea to forecast costs against estimated player counts well before launch to make sure servers can add more value than what they cost. You’ll need to keep evaluating this in the months and years after launch, and it could even be worth thinking about a contingency in case you end up having to turn them off (e.g., you could switch back to P2P).

If you already have a CI pipeline in place, you can automate the uploading of game servers along with client builds. Having the option to spin up a fleet to test nightly builds can be pretty useful, though you’ll need to keep an eye on your host’s storage space limits, etc.

Choosing which regions to host servers in can be a difficult balance of cost versus demand, particularly before launch. Both AWS and Google Cloud have pretty good worldwide coverage, and AWS recently opened up GameLift in China, too. If you’re passing player latency data up to your matchmaker, then you’ll be able to more clearly see where your player demand is coming from, and which regions could do with some servers.

You may also want to overprovision for launch to ensure any “best case” traffic is handled. Support will also be important come launch day, since you’ll no doubt have something go wrong and need as much as help as possible to put out fires - something I’m probably doing right now!

Matchmaking

Once you have a scalable fleet of dedicated servers waiting to receive games, you’ll need a way to actually match players together. Although most platforms have some kind of first-party matchmaking API, to do cross-play, you’ll need a centralized matchmaker that can also talk to your server manager.

Like with hosting, there are many different options for game backends, ranging from fully integrated solutions with hosting/matchmaking/client-side SDKs, to providers that handle servers/matchmaking but still require your own backend to handle auth and client communication.

At bare minimum, a client needs to be able to tell a backend that they want to find a match with certain parameters. The backend will then authenticate their request using each platform’s auth token, and add them to the matchmaking pool. Once a match has been found, the matchmaker then needs to assign the game session to a server, tell each client about that server’s IP and port, and tell the server which players it should allow to join, as well as any useful matchmaking data, such as teams or game type.

When evaluating different options, try to think not only about your current requirements but also about future possibilities to avoid denying yourself features that would’ve been great to add down the line. For instance:

• Do you want rule-based or lobby-based matchmaking?
• Do you want to be able to relax rules over time?
• Do you want to take latency information into account?
• Do you want to allow queueing for multiple game types at once?
• Do you want to support parties in matchmaking?
• Can they be cross-platform parties?
• Do you want to support multiple local players?
• Do you want teams, and can parties team up?
• Do you want asynchronous matchmaking, so that players can do something else in the game while waiting?
• Do you need to be able to restrict which platforms can be matched together? (hopefully, this is eventually a “no!”)
• Do you need avoid- or block-lists?
• Do you need access to persistent player data to enable skill-based matchmaking and/or ranked modes?
• Do you want to support back-fill, so that someone can join a match in-progress?

Any specific advice we can give will probably quickly become outdated, but one timeless recommendation is to start thinking about this as early as possible.

When evaluating matchmaking features, also look at each backend provider’s other offerings, such as player identity, friends, presence, parties, leaderboards, analytics, and more. If you do end up using a backend provider, the quality of these other services could help split the difference — and they’re all things that take time to integrate well into your game. You don’t want to have to rush into something that could be hard to change later! That being said, if you’re already using a certain provider for some areas but want to use someone else for matchmaking, then you can do that, too. Online services seem to inevitably involve a number of moving parts, and it’s up to you which parts you source from where.

Versioning and net compatibility

Once you have dedicated servers — and especially once you have cross-play — you need to think about client/server versioning, and how that affects your matchmaking. This is actually one area that I haven’t seen many nicely integrated solutions for, so you may have to come up with something yourself.

What happens when you need to release an update that breaks net compatibility? Can you release new, net-compatible server builds without having to update clients? If you do need to break cross-play for a certain platform, can you still require other platforms to be on a certain version?

It can take some time for a client update to roll out across all platforms and regions, so you probably don’t want to just shut down old servers immediately in case a player doesn’t even have the update available to them yet. And even once the update does become available to someone, you wouldn’t want to kick them in the middle of a match.

Creating a seamless update experience isn’t necessarily hard, though it does require some forethought and planning. We settled on a setup that accepts matches for both old and new versions during the update window, keeping the old servers around so players can finish their match. We allow clients to check for updates themselves (and apply an update before starting new matches), and after a reasonable propagation time, we turn off the old servers and reject requests from any old clients. If we need to deploy a server-only patch, the matchmaker will direct any new matches to the new patch’s fleets, and then once all matches on the old fleets have been completed, we can take them down in the background.

Testing your client/server builds for net compatibility is also important, particularly when making fixes on an existing release branch. If you use the Blueprint Nativization feature, you may also run into compatibility issues when connecting to a nativized server from a non-nativized client (e.g., PIE or cook on the fly). If you want to avoid being kicked with a “Net GUID Mismatch” error while testing in-editor, you can set the net.IgnoreNetworkChecksumMismatch CVar to 1.

A related tip is to check out the TimeoutMultiplierForUnoptimizedBuilds config variable, which can increase the net driver’s timeout thresholds in-editor. Since we have relatively short connection timeouts and “lagging” thresholds, our in-editor testing could often time out during travel if something took longer than expected to load or compile. If making a networked game, you no doubt already use multiple PIE windows and multi-process PIE, but I think it’s worth acknowledging just how awesome these features are compared to what’s available in other engines.

Platform requirements

As you will have learned from cross-platform development, there can be some pretty specific requirements placed on you by each platform, especially if your game is online. In terms of cross-play, this can involve things like indicating (or not indicating) which network other players are on, sanitizing player names and handling collisions, validating player privileges and calling other platform-specific APIs.

It’s best to talk with each platform early on to make sure you have all the information needed to implement these requirements and avoid nasty surprises that can leave a dent in your schedule.

Consider cross-play!

Indie multiplayer can be tough, but cross-play is one way to help grow your matchmaking pool and increase the return on your cross-platform investment. It may seem like a lot of work for an independent studio, but it can be a big selling point for your game, and it’ll only become easier as game service providers mature and platforms become more open.

Hopefully this has been a helpful overview of what to consider when adding cross-play to your own game!

Hyper Jam is available now on Steam, PS4, and Xbox One!