Most of the information on this page was gleamed form this video that Frontier did at the Amazon re:Invent conference: https://www.youtube.com/watch?v=EvJPyjmfdz0&feature=youtu.be

The multiplayer experience is handled by two main components - the raw peer-to-peer real-time multi-commander interactions (dog fights etc), and the intermediate edServer than handles bringing players together in the same instance.

“edServer” is the name of the main central servers that players first connect to when playing Elite Dangerous. The edServers monitor all player movements around the galaxy (e.g. which systems each player is in and so on) and when different players start to get close to each other, the edServer instructs the players to establish a peer-to-peer connection with each other to enable real-time multiplayer.

The edServer that you connect to will vary - each edServer will hold many open sessions for different players at the same time. The rationale given for this is so that when a player connects to the sever, an internal lookup can be made to find the appropriate edServer that has players who are a good match (e.g. are in the same star system, potentially other criteria), and then connect that player to that edServer so that there are more players to interact with.

There is not a single edServer for everyone - there are potentially dozens of edServer instances running globally at any one time depending on player load and scaling needs. The shared galaxy state is stored in various different data stores, and accessed simultaneously by the different edServers as required. So far we know that Elite Dangerous uses:

• Amazon Dynamo DB
• Redis (“We love redis”)
• Mongo DB (“We quite like Mongo too”)
• Amazon S3
• MySQL

Access to this shared galaxy state is via a RESTful API written in PHP and automatically scaled by Amazon's Elastic Beanstalk. Frontier Developments use PHP Composer dependency management system, using a private dependency cache within their Amazon cloud.

Auth for the API is handled by HTTP headers, with authentication handed off to external platforms as appropriate (e.g. XBox platform for console users, Frontier Development's servers for PC & Mac).

Each AWS C3 EC2 instance that is being used to run an edServer instance on also runs a watchDog.py script that watches a S3 bucket looking for new releases.

If a new release is found, watchDog.py auto-shutsdown the server and redploys the new build.

The edServers have a concept of server frames per second, or server FPS. The server FPS is used to identify when an edServer is overloaded and another needs to be created (see Scaling section below). It reflects how fast the edServer is able to run its main loop, as explained by hchalkley on the forums:

“the server runs a loop, processing incoming packets, sending out packets, and updating state: typically repeating between around fifty and five hundred times a second, depending on load. It's a similar loop to the game, it just doesn't have to render anything, or wait for a frame sync.”

The server FPS does not affect the FPS you see on your computer.

Each edServer instance also runs a metrics.py script that reports players count and server FPS to Amazon CloudWatch. Stats are aggregated by region to see a moving 5 minute average load in any region.

There is a very strong correlation between players & server FPS. When the average server FPS gets too low, a new server is automatically created in AWS to bring the average FPS up in a given region.

edServers are not terminated if there are players on them! Players are “stacked” onto each server so that each edServer is “filled up” to capacity, rather than having players distributed evenly across all available edServers.

This has the result of matching the player experience better (in a matchmaking sense - i.e. that there are lots of players on the server to meet out in space), and easier to terminate idle servers since they want to terminate empty servers only.

The peer-to-peer (aka P2P) connections are direct UDP connections using a Frontier Developments 's in-house UDP protocol for communication of fast, real-time “twitch” gameplay traffic (i.e. ship movement and direction, firing lasers and so on).

Peers/players can connect to several players at once via this UDP peer-to-peer approach - currently we don't know the upper limit but it would make sense that this is the upper-limit of a single Instance.

From what we understand so far, it would seem that you can only establish a peer-to-peer connection to other players who are also on the same edServer instance, since a lot of emphasis was placed on making sure that servers are on an instance that was a good match for them (e.g. with lots of other player sessions that were in the same system etc).

Each availability zone has its own MongoDB instance & background processor with an autoscaling group for the PHP running the actual API calls.

Shared DynamoDB, S3, MySQL and Redis ElastiCache between availability zones. “Loads and loads of redis. Redis is great.”

background processor runs on a single EC2 instance on a couple of CPUs.

Basic network services.

Latency to edServer is not super-critical from peers, but obviously p2p in latency critical.

p2p often behind firewalls or NAT - FD uses an open source TURN server on linux instances in global AWS availability zones (“one in Ireland, one in Australia - we have some others dotted around as well”) to provide STUN & TURN services at minimal latency to each peer.

Uses Amazon Route 53 to latency-based routing to route peer to closest (on the internet, not geographically) TURN server.

Auto-scales the number of STUN/TURN servers based on load from the Route 53 service.

Commanders and ships stored in RDMS (MySQL) and Document-based DB (MongoDB) respectively. Frontier use a custom MongoDB 3 cluster.

Game-play critical stats like credits, mission completion etc goes into the MySQL database. Although the cost of handling RDMS reads and writes are high, the ACID nature of MySQL is critical to ensure that the player's data is not corrupted or lost (e.g. that if a player spends 1,000,000 to get a ship, they have to make sure that the money is deducted AND they get a ship as one transaction without the risk of the player losing the money and getting no ship, or getting the ship and not spending the money!)

Ship data (e.g. what modules installed, fuel available etc) would be expensive to store in a RDMS so a document model is used in MongoDB.

Elite Dangerous only stores “interesting data” - i.e. something a player has interacted with and where the data has moved from its “default state” or “default value”.

Example of trading and how it influences the “default value”:

• default stock of 1000 units of fish
• player A buys 200 units of fish
• API stores a modified from the default unit of 0.8
• next player comes along and gets the modifier of the default value
• once the background sim repopulates the stock back to its default state, the multiplier is not set to 1.0, but is simply deleted and its as if nothing happened.

Background processor gets a periodic summary of all items bought and sold from MongoDB and generates the modifiers for all clients to use (i.e. if it knows that there were 200 fish bought at Sol, 100 sold, then the net is -100 so modifier ends up at 0.9 etc) to generate an “almost real time” system. This information is also used each day to genernate live trade routes on the maps.

Kabana used for in-game analytics by reading the transaction logs from Amazon SQS to an on-premise Elasticsearch server. Lets FD work out what people are doing (drilling down etc) but also see trends and work out generally what people are doing, and if there are any outliers that are not supposed to be happening.

StellarForge generates a 1:1 scale model of the galaxy - stats including:

class
mass
radius (in "earths")
surface temperature (K)
block body temperature (K)
magma temperature (K)
semi-major axis (AU)
eccentricity
inclination (radians)
surface pressure (Atm)
valcanism type (eg. "Water Geysers)")
Atmospheric composition (class-based, e.g. "earth like", breakdown on ratios of difference gasses)

Stellar forge data is cached in MySQL, but sometimes they need to use the sophisticated simulation software to get 100% consistent data.

First discovery credit stored in Amazon S3 on a per-body basis (name of body, commander name, discovered date). DyanamDB is used for the ship's navigation computers - originally used MongoDB but scalability was tough so they switched.

Uses Amazon SQS to log actions (e.g. kills a player, trades, generates a chunk of asteroid form mining). Background processor reads items off of the queue and stores entries into the MongoDB instances.

action consequencies added to “buckets” per faction (influence, boom, bust, famine etc) stored in MongoDB. Approx 100,000 factions.

Background processor reads in bucket values and does things (e.g. change trade data, or sets up conflict zones between factions if one faction got powerful etc)

MySQL stores data for community goals since it is generally batch-based (i.e. players trade community goal items one cargo-hold at a time etc) and there are only a few community goals active at once so RDMS can handle load in these situations. Data is also written into Redis and is used for leader boards for community goals as well via Sorted Sets (leader boards re-built from MySQL or snapshot if there is a new redis instance)

Writen in PHP & uses Drupal & Magento. MySQL instance stores data. Launcher served directly from S3 using a per-customer URL. Game builds are stored in S3 and fronted by Amazon CloudFront.

Launcher connects to FD authorisation servers that tells launcher which products the customer is authorised for (e.g. main game, betas etc). Main game client also checks in with the auth servers during play (also gets things like what rewards are they able to get - e.g. kickstarter backers get extra stuff etc)

Custom-built matchmaking for PC, XBox uses standard platform matchmaking system.

Matchmaking using PHP API & Redis queues - people join the Redis queue and the background processor picks people off of the queue as appropriate (e.g. what game do they want to play etc), then groups them into lobbies.