Agones – Guides

Docs: Feature Stages

Thu, 26 Sep 2019 01:20:41 +0000

Supported Versions

Agones versions are expressed as x.y.z, where x is the major version, y is the minor version, and z is the patch version following Semantic Versioning terminology.

Agones Features

A feature within Agones can be in Alpha, Beta or Stable stage.

Feature Gates

Alpha and Beta features can be enabled or disabled through the agones.featureGates configuration option that can be found in the Helm configuration documentation.

The current set of alpha and beta feature gates:

Feature Name	Gate	Default	Stage	Since
Allocated GameServers are notified on relevant Fleet Updates	`FleetAllocationOverflow`	Enabled	`Beta`	1.37.0
CountsAndLists	`CountsAndLists`	Disabled	`Alpha`	1.37.0
DisableResyncOnSDKServer	`DisableResyncOnSDKServer`	Disabled	`Alpha`	1.37.0
GameServer player capacity filtering on GameServerAllocations	`PlayerAllocationFilter`	Disabled	`Alpha`	1.14.0
Player Tracking	`PlayerTracking`	Disabled	`Alpha`	1.6.0
Example Gate (not in use)	`Example`	Disabled	None	0.13.0

Note

If you aren’t sure if Feature Flags have been set correctly, have a look at the The Feature Flag I enabled/disabled isn’t working as expected troubleshooting section.

Description of Stages

Alpha

An Alpha feature means:

Disabled by default.
Might be buggy. Enabling the feature may expose bugs.
Support for this feature may be dropped at any time without notice.
The API may change in incompatible ways in a later software release without notice.
Recommended for use only in short-lived testing clusters, due to increased risk of bugs and lack of long-term support.

Note

Please do try Alpha features and give feedback on them. This is important to ensure less breaking changes through the Beta period.

Beta

A Beta feature means:

Enabled by default, but able to be disabled through a feature gate.
The feature is well tested. Enabling the feature is considered safe.
Support for the overall feature will not be dropped, though details may change.
The schema and/or semantics of objects may change in incompatible ways in a subsequent beta or stable releases. When this happens, we will provide instructions for migrating to the next version. This may require deleting, editing, and re-creating API objects. The editing process may require some thought. This may require downtime for applications that rely on the feature.
Recommended for only non-business-critical uses because of potential for incompatible changes in subsequent releases. If you have multiple clusters that can be upgraded independently, you may be able to relax this restriction.

Note

Note: Please do try Beta features and give feedback on them! After they exit beta, it may not be practical for us to make more changes.

Stable

A Stable feature means:

The feature is enabled and the corresponding feature gate no longer exists.
Stable versions of features will appear in released software for many subsequent versions.

Feature Stage Indicators

There are a variety of features with Agones, how can we determine what stage each feature is in?

Below are indicators for each type of functionality that can be used to determine the feature stage for a given aspect of Agones.

Custom Resource Definitions (CRDs)

This refers to Kubernetes resource for Agones, such as GameServer, Fleet and GameServerAllocation.

New CRDs

For new resources, the stage of the resource will be indicated by the apiVersion of the resource.

For example: apiVersion: "agones.dev/v1" is a stable resource, apiVersion: "agones.dev/v1beta1" is a beta stage resource, and apiVersion: "agones.dev/v1alpha1" is an alpha stage resource.

New CRD attributes

When alpha and beta attributes are added to an existing stable Agones CRD, we will follow the Kubernetes Adding Unstable Features to Stable Versions Guide to optimise on the least amount of breaking changes for users as attributes progress through feature stages.

alpha and beta attributes will be added to the existing CRD as optional and documented with their feature stage. Attempting to populate these alpha and beta attributes on an Agones CRD will return a validation error if their accompanying Feature Flag is not enabled.

alpha and beta attributes can be subject to change of name and structure, and will result in breaking changes before moving to a stable stage. These changes will be outlined in release notes and feature documentation.

Agones Game Server SDK

Any alpha or beta Game Server SDK functionality will be a subpackage of the sdk package. For example , functionality found in a sdk.alphav1 package should be considered at the alpha feature stage.

Only experimental functionality will be found in any alpha and beta SDK packages, and as such may change as development occurs.

As SDK features move to through feature stages towards stable, the previous version of the SDK API will remain for at least one release to enable easy migration to the more stable feature stage (i.e. from alpha -> beta, beta -> stable)

Any other SDK functionality not marked as alpha or beta is assumed to be stable.

REST & gRPC APIs

REST and gRPC API will have versioned paths where appropriate to indicate their feature stage.

For example, a REST API with a prefix of v1alpha1 is an alpha stage feature: http://api.example.com/v1alpha1/exampleaction.

Similar to the SDK, any alpha or beta gRPC functionality will be a subpackage of the main API package. For example, functionality found in a api.alphav1 package should be considered at the alpha feature stage.

Docs: Best Practices

Fri, 12 May 2023 00:00:00 +0000

Overview

Running Agones in production takes consideration, from planning your launch to figuring out the best course of action for cluster and Agones upgrades. On this page, we’ve collected some general best practices. We also have cloud specific pages for:

Google Kubernetes Engine (GKE)

If you are interested in submitting best practices for your cloud prodiver / on-prem, please contribute!

Separation of Agones from GameServer nodes

When running in production, Agones should be scheduled on a dedicated pool of nodes, distinct from where Game Servers are scheduled for better isolation and resiliency. By default Agones prefers to be scheduled on nodes labeled with agones.dev/agones-system=true and tolerates the node taint agones.dev/agones-system=true:NoExecute. If no dedicated nodes are available, Agones will run on regular nodes. See taints and tolerations for more information about Kubernetes taints and tolerations.

If you are collecting Metrics using our standard Prometheus installation, see the installation guide for instructions on configuring a separate node pool for the agones.dev/agones-metrics=true taint.

See Creating a Cluster for initial set up on your cloud provider.

Redundant Clusters

Allocate Across Clusters

Agones supports Multi-cluster Allocation, allowing you to allocate from a set of clusters, versus a single point of potential failure. There are several other options for multi-cluster allocation:

Anthos Service Mesh can be used to route allocation traffic to different clusters based on arbitrary criteria. See Global Multiplayer Demo for an example where the match maker influences which cluster the allocation is routed to.
Allocation Endpoint can be used in Cloud Run to proxy allocation requests.
Or peruse the Third Party Examples

Spread

You should consider spreading your game servers in two ways:

Across geographic fault domains (GCP regions, AWS availability zones, separate datacenters, etc.): This is desirable for geographic fault isolation, but also for optimizing client latency to the game server.
Within a fault domain: Kubernetes Clusters are single points of failure. A single misconfigured RBAC rule, an overloaded Kubernetes Control Plane, etc. can prevent new game server allocations, or worse, disrupt existing sessions. Running multiple clusters within a fault domain also allows for easier upgrades.

Docs: Agones Game Server Client SDKs

Wed, 02 Jan 2019 10:16:30 +0000

Overview

The client SDKs are required for a game server to work with Agones.

The current supported SDKs are:

You can also find some externally supported SDKs in our Third Party Content.

The SDKs are relatively thin wrappers around gRPC generated clients, or an implementation of the REST API (exposed via grpc-gateway), where gRPC client generation and compilation isn’t well supported.

They connect to a small process that Agones coordinates to run alongside the Game Server in a Kubernetes Pod. This means that more languages can be supported in the future with minimal effort (but pull requests are welcome! 😊 ).

There is also local development tooling for working against the SDK locally, without having to spin up an entire Kubernetes infrastructure.

Connecting to the SDK Server

Starting with Agones 1.1.0, the port that the SDK Server listens on for incoming gRPC or HTTP requests is configurable. This provides flexibility in cases where the default port conflicts with a port that is needed by the game server.

Agones will automatically set the following environment variables on all game server containers:

AGONES_SDK_GRPC_PORT: The port where the gRPC server is listening (defaults to 9357)
AGONES_SDK_HTTP_PORT: The port where the grpc-gateway is listening (defaults to 9358)

The SDKs will automatically discover and connect to the gRPC port specified in the environment variable.

If your game server requires using a REST client, it is advised to use the port from the environment variable, otherwise your game server will not be able to contact the SDK server if it is configured to use a non-default port.

Function Reference

While each of the SDKs are canonical to their languages, they all have the following functions that implement the core responsibilities of the SDK.

For language specific documentation, have a look at the respective source (linked above), and the examples .

Calling any of state changing functions mentioned below does not guarantee that GameServer Custom Resource object would actually change its state right after the call. For instance, it could be moved to the Shutdown state elsewhere (for example, when a fleet scales down), which leads to no changes in GameServer object. You can verify the result of this call by waiting for the desired state in a callback to WatchGameServer() function.

Functions which changes GameServer state or settings are:

Ready()
Shutdown()
SetLabel()
SetAnnotation()
Allocate()
Reserve()
Alpha().SetCapacity()
Alpha().PlayerConnect()
Alpha().PlayerDisconnect()
Alpha().SetCounterCount()
Alpha().IncrementCounter()
Alpha().DecrementCounter()
Alpha().SetCounterCapacity()
Alpha().AppendListValue()
Alpha().DeleteListValue()
Alpha().SetListCapacity()

Lifecycle Management

Ready()

This tells Agones that the Game Server is ready to take player connections. Once a Game Server has specified that it is Ready, then the Kubernetes GameServer record will be moved to the Ready state, and the details for its public address and connection port will be populated.

While Agones prefers that Shutdown() is run once a game has completed to delete the GameServer instance, if you want or need to move an Allocated GameServer back to Ready to be reused, you can call this SDK method again to do this.

Health()

This sends a single ping to designate that the Game Server is alive and healthy. Failure to send pings within the configured thresholds will result in the GameServer being marked as Unhealthy.

See the gameserver.yaml for all health checking configurations.

Reserve(seconds)

With some matchmaking scenarios and systems it is important to be able to ensure that a GameServer is unable to be deleted, but doesn’t trigger a FleetAutoscaler scale up. This is where Reserve(seconds) is useful.

Reserve(seconds) will move the GameServer into the Reserved state for the specified number of seconds (0 is forever), and then it will be moved back to Ready state. While in Reserved state, the GameServer will not be deleted on scale down or Fleet update, and also it could not be Allocated using GameServerAllocation.

This is often used when a game server process must register itself with an external system, such as a matchmaker, that requires it to designate itself as available for a game session for a certain period. Once a game session has started, it should call SDK.Allocate() to designate that players are currently active on it.

Calling other state changing SDK commands such as Ready or Allocate will turn off the timer to reset the GameServer back to the Ready state or to promote it to an Allocated state accordingly.

Allocate()

With some matchmakers and game matching strategies, it can be important for game servers to mark themselves as Allocated. For those scenarios, this SDK functionality exists.

There is a chance that GameServer does not actually become Allocated after this call. Please refer to the general note in Function Reference above.

The agones.dev/last-allocated annotation will be set on the GameServer to an RFC3339 formatted timestamp of the time of allocation, even if the GameServer was already in an Allocated state.

Note that if using SDK.Allocate() in combination with GameServerAllocations, it’s possible for the agones.dev/last-allocated timestamp to move backwards if clocks are not synchronized between the Agones controller and the GameServer pod.

Note

Using a GameServerAllocation is preferred in all other scenarios, as it gives Agones control over how packed GameServers are scheduled within a cluster, whereas with Allocate() you relinquish control to an external service which likely doesn’t have as much information as Agones.

Shutdown()

This tells Agones to shut down the currently running game server. The GameServer state will be set Shutdown and the backing Pod will be Terminated.

It’s worth reading the Termination of Pods Kubernetes documentation, to understand the termination process, and the related configuration options.

As a rule of thumb, implement a graceful shutdown in your game sever process when it receives the TERM signal from Kubernetes when the backing Pod goes into Termination state.

Be aware that if you use a variation of System.exit(0) after calling SDK.Shutdown(), your game server container may restart for a brief period, inline with our Health Checking policies.

If the SDK server receives a TERM signal before calling SDK.Shutdown(), the SDK server will stay alive for the period of the terminationGracePeriodSeconds until SDK.Shutdown() has been called.

Configuration Retrieval

GameServer()

This returns most of the backing GameServer configuration and Status. This can be useful for instances where you may want to know Health check configuration, or the IP and Port the GameServer is currently allocated to.

Since the GameServer contains an entire PodTemplate the returned object is limited to that configuration that was deemed useful. If there are areas that you feel are missing, please file an issue or pull request.

The easiest way to see what is exposed, is to check the sdk.proto , specifically at the message GameServer.

For language specific documentation, have a look at the respective source (linked above), and the examples .

WatchGameServer(function(gameserver){…})

This executes the passed in callback with the current GameServer details whenever the underlying GameServer configuration is updated. This can be useful to track GameServer > Status > State changes, metadata changes, such as labels and annotations, and more.

In combination with this SDK, manipulating Annotations and Labels can also be a useful way to communicate information through to running game server processes from outside those processes. This is especially useful when combined with GameServerAllocation applied metadata.

The easiest way to see what is exposed, is to check the sdk.proto , specifically at the message GameServer.

For language specific documentation, have a look at the respective source (linked above), and the examples .

Metadata Management

SetLabel(key, value)

This will set a Label value on the backing GameServer record that is stored in Kubernetes.

To maintain isolation, the key value is automatically prefixed with the value “agones.dev/sdk-”. This is done for two main reasons:

The prefix allows the developer to always know if they are accessing or reading a value that could have come, or may be changed by the client SDK. Much like private vs public scope in a programming language, the Agones SDK only gives you access to write to part of the set of labels and annotations that exist on a GameServer.
The prefix allows for a smaller attack surface if the GameServer container gets compromised. Since the game container is generally externally exposed, and the Agones project doesn’t control the binary that is run within it, limiting exposure if the game server becomes compromised is worth the extra development friction that comes with having this prefix in place.

Warning

There are limits on the characters that be used for label keys and values. Details are here.

You will need to take them into account when combined with the label prefix above.

Setting GameServer labels can be useful if you want information from your running game server process to be observable or searchable through the Kubernetes API.

SetAnnotation(key, value)

This will set an Annotation value on the backing GameServer record that is stored in Kubernetes.

To maintain isolation, the key value is automatically prefixed with “agones.dev/sdk-” for the same reasons as in SetLabel(…) above. The isolation is also important as Agones uses annotations on the GameServer as part of its internal processing.

Setting GameServer annotations can be useful if you want information from your running game server process to be observable through the Kubernetes API.

Counters And Lists

Warning

The Counters And Lists feature is currently Alpha, not enabled by default, and may change in the future.

Use the FeatureGate CountsAndLists to enable and test this feature.

See the Feature Gate documentation for details on how to enable features.

The Counters and Lists features in the SDK offer a flexible configuration for tracking various entities like players, rooms, and sessions.

Declared keys and default values for Counters and Lists are specified in GameServer.Spec.Counters and GameServer.Spec.Lists respectively.

Modified Counter and List values and capacities will be updated in GameServer.Status.Counters and GameServer.Status.Lists respectively.

Note

The SDK batches mutation operations every 1 second for performance reasons. However, changes made and subsequently retrieved through the SDK will be atomically accurate through the SDK, as those values are tracked within the SDK Server sidecar process.

Changes made through Allocation or the Kubernetes API to GameServer.Spec.Counters and GameServer.Spec.Lists will be eventually consistent when being retrieved through the SDK.

Since the Agones SDK server batches the update operations of GameServer.Status.Counters and GameServer.Status.Lists asynchronously, this means that if you update GameServer.status values through both the SDK and the Allocation/Kubernetes API, the batch processing may silently truncate some of those values to the capacity of that Counter or List.

Counters

All functions will return an error if the specified key is not predefined in the GameServer.Spec.Counters resource configuration.

Note: For Counters, the default setting for the capacity is preset to 1000. It is recommended to avoid configuring the capacity to max(int64), as doing so could cause problems with JSON Patch operations.

Alpha().GetCounterCount(key)

This function retrieves either the GameServer.Status.Counters[key].Count or the SDK awaiting-batch value for a given key, whichever is most up to date.

Alpha().SetCounterCount(key, amount)

This function sets the value of GameServer.Status.Counters[key].Count for the given key to the passed in amount. This operation overwrites any previous values and the new value cannot exceed the Counter’s capacity.

Alpha().IncrementCounter(key, amount)

This function increments GameServer.Status.Counters[key].Count for the given key by the passed in non-negative amount. The function returns an error if the Counter is already at capacity (at time of operation), indicating no increment will occur.

Alpha().DecrementCounter(key, amount)

This function decreases GameServer.Status.Counters[key].Count for the given key by the passed in non-negative amount. It returns an error if the Counter’s count is already at zero.

Alpha().SetCounterCapacity(key, amount)

This function sets the maximum GameServer.Status.Counters[key].Capacity for the given key by the passed in non-negative amount. A capacity value of 0 indicates no capacity limit.

Alpha().GetCounterCapacity(key)

This function retrieves either the GameServer.Status.Counters[key].Capacity or the SDK awaiting-batch value for the given key, whichever is most up to date.

Lists

All functions will return an error if the specified key is not predefined in the GameServer.Spec.Lists resource configuration.

Alpha().AppendListValue(key, value)

This function appends the specified string value to the List in GameServer.Status.Lists[key].Values.

An error is returned if the string already exists in the list or if the list is at capacity.

Alpha().DeleteListValue(key, value)

This function removes the specified string value from the List in GameServer.Status.Lists[key].Values.

An error is returned if the string does not exist in the list.

Alpha().SetListCapacity(key, amount)

This function sets the maximum capacity for the List at GameServer.Status.Lists[key].Capacity.

The capacity value is required to be between 0 and 1000.

Alpha().GetListCapacity(key)

This function retrieves either the GameServer.Status.Lists[key].Capacity or the SDK awaiting-batch value for the given key, whichever is most up to date.

Alpha().GetListValues(key)

This function retrieves either the GameServer.Status.Lists[key].Values or the SDK awaiting-batch values array for the given key, whichever is most up to date.

Alpha().ListContains(key, value)

Convenience function, which returns if the specific string value exists in the results of Alpha().GetListValues(key).

Alpha().GetListLength(key)

Convenience function, which retrieves the length of the results of Alpha().GetListValues(key).

Player Tracking

Counters and Lists will eventually replace the Alpha functionality of Player Tracking, which will subsequently be removed from Agones.

If you are currently using this Alpha feature, we would love for you to test (and ideally migrate to!) this new functionality to Counters and Lists to ensure it meet all your needs.

Warning

The Player Tracking feature is currently Alpha, not enabled by default, and may change in the future.

Use the FeatureGate PlayerTracking to enable and test this feature.

See the Feature Gate documentation for details on how to enable features.

Alpha().PlayerConnect(playerID)

This function increases the SDK’s stored player count by one, and appends this playerID to GameServer.Status.Players.IDs.

GameServer.Status.Players.Count and GameServer.Status.Players.IDs are then set to update the player count and id list a second from now, unless there is already an update pending, in which case the update joins that batch operation.

PlayerConnect() returns true and adds the playerID to the list of playerIDs if this playerID was not already in the list of connected playerIDs.

If the playerID exists within the list of connected playerIDs, PlayerConnect() will return false, and the list of connected playerIDs will be left unchanged.

An error will be returned if the playerID was not already in the list of connected playerIDs but the player capacity for the server has been reached. The playerID will not be added to the list of playerIDs.

Note

Do not use this method if you are manually managing GameServer.Status.Players.IDs and GameServer.Status.Players.Count through the Kubernetes API, as indeterminate results will occur.

Alpha().PlayerDisconnect(playerID)

This function decreases the SDK’s stored player count by one, and removes the playerID from GameServer.Status.Players.IDs.

PlayerDisconnect() will return true and remove the supplied playerID from the list of connected playerIDs if the playerID value exists within the list.

If the playerID was not in the list of connected playerIDs, the call will return false, and the connected playerID list will be left unchanged.

Note

Do not use this method if you are manually managing GameServer.Status.Players.IDs and GameServer.Status.Players.Count through the Kubernetes API, as indeterminate results will occur.

Alpha().SetPlayerCapacity(count)

Update the GameServer.Status.Players.Capacity value with a new capacity.

Alpha().GetPlayerCapacity()

This function retrieves the current player capacity. This is always accurate from what has been set through this SDK, even if the value has yet to be updated on the GameServer status resource.

Note

If GameServer.Status.Players.Capacity is set manually through the Kubernetes API, use SDK.GameServer() or SDK.WatchGameServer() instead to view this value.

Alpha().GetPlayerCount()

This function retrieves the current player count. This is always accurate from what has been set through this SDK, even if the value has yet to be updated on the GameServer status resource.

Note

If GameServer.Status.Players.IDs is set manually through the Kubernetes API, use SDK.GameServer() or SDK.WatchGameServer() instead to retrieve the current player count.

Alpha().IsPlayerConnected(playerID)

This function returns if the playerID is currently connected to the GameServer. This is always accurate from what has been set through this SDK, even if the value has yet to be updated on the GameServer status resource.

Note

If GameServer.Status.Players.IDs is set manually through the Kubernetes API, use SDK.GameServer() or SDK.WatchGameServer() instead to determine connected status.

Alpha().GetConnectedPlayers()

This function returns the list of the currently connected player ids. This is always accurate from what has been set through this SDK, even if the value has yet to be updated on the GameServer status resource.

Note

If GameServer.Status.Players.IDs is set manually through the Kubernetes API, use SDK.GameServer() or SDK.WatchGameServer() instead to list the connected players.

Writing your own SDK

If there isn’t an SDK for the language and platform you are looking for, you have several options:

gRPC Client Generation

If client generation is well supported by gRPC, then generate client(s) from the proto files found in the proto/sdk , directory and look at the current sdks to see how the wrappers are implemented to make interaction with the SDK server simpler for the user.

REST API Implementation

If client generation is not well supported by gRPC, or if there are other complicating factors, implement the SDK through the REST HTTP+JSON interface. This could be written by hand, or potentially generated from the Swagger/OpenAPI Specifications .

Finally, if you build something that would be usable by the community, please submit a pull request!

SDK Conformance Test

There is a tool SDK server Conformance checker which will run Local SDK server and record all requests your client is performing.

In order to check that SDK is working properly you should write simple SDK test client which would use all methods of your SDK.

Also to test that SDK client is receiving valid Gameserver data, your binary should set the same Label value as creation timestamp which you will receive as a result of GameServer() call and Annotation value same as gameserver UID received by Watch gameserver callback.

Complete list of endpoints which should be called by your test client is the following:

ready,allocate,setlabel,setannotation,gameserver,health,shutdown,watch

In order to run this test SDK server locally use:

SECONDS=30 make run-sdk-conformance-local

Docker container would timeout in 30 seconds and give your the comparison of received requests and expected requests

For instance you could run Go SDK conformance test and see how the process goes:

SDK_FOLDER=go make run-sdk-conformance-test

In order to add test client for your SDK, write sdktest.sh and Dockerfile. Refer to Golang SDK Conformance testing directory structure .

Building the Tools

If you wish to build the binaries from source the make target build-agones-sdk-binary will compile the necessary binaries for all supported operating systems (64 bit windows, linux and osx).

You can find the binaries in the bin folder in `cmd/sdk-server` once compilation is complete.

See Developing, Testing and Building Agones for more details.

Docs: Windows Gameservers

Mon, 19 Apr 2021 21:14:19 +0000

Warning

Running GameServers on Windows nodes is currently Alpha, and any feedback would be appreciated.

Prerequisites

The following prerequisites are required to create a GameServer:

A Kubernetes cluster with the UDP port range 7000-8000 open on each node.
Agones controller installed in the targeted cluster
kubectl properly configured
Netcat which is already installed on most Linux/macOS distributions, for windows you can use WSL.

If you don’t have a Kubernetes cluster you can follow these instructions to create a cluster on Google Kubernetes Engine (GKE), Minikube or Azure Kubernetes Service (AKS), and install Agones.

For the purpose of this guide we’re going to use the simple-game-server example as the GameServer container. This example is a very simple UDP server written in Go. Don’t hesitate to look at the code of this example for more information.

Ensure that you have some nodes to your cluster that are running Windows.

Objectives

Create a GameServer on a Windows node.
Connect to the GameServer.

1. Create a GameServer

Note

Starting with version 0.3, the simple-game-server example is compiled as a multi-arch docker image that will run on both Linux and Windows. To ensure that the game server runs on a Windows node, a nodeSelector of "kubernetes.io/os": windows must be added to the game server specification.

Create a GameServer using the following command:

kubectl create -f https://raw.githubusercontent.com/googleforgames/agones/release-1.39.0/examples/simple-game-server/gameserver-windows.yaml

You should see a successful output similar to this:

gameserver.agones.dev/simple-game-server-4ss4j created

Verify that the GameServer becomes Ready by running:

kubectl get gameservers

It should look something like this:

NAME                       STATE     ADDRESS       PORT   NODE     AGE
simple-game-server-7pjrq   Ready   35.233.183.43   7190   agones   3m

Take a note of the Game Server IP address and ports.

For the full details of the YAML file head to the GameServer Specification Guide

2. Connect to the GameServer

Note

If you have Agones installed on Google Kubernetes Engine, and are using Cloud Shell for your terminal, UDP is blocked. For this step, we recommend SSH’ing into a running VM in your project, such as a Kubernetes node. You can click the ‘SSH’ button on the Google Compute Engine Instances page to do this. Run toolbox on GKE Node to run docker container with tools and then nc command would be available.

You can now communicate with the Game Server:

Note

If you do not have netcat installed (i.e. you get a response of nc: command not found), you can install netcat by running sudo apt install netcat.

If you are on Windows, you can alternatively install netcat on WSL, or download a version of netcat for Windows from nmap.org.

nc -u {IP} {PORT}
Hello World !
ACK: Hello World !
EXIT

You can finally type EXIT which tells the SDK to run the Shutdown command, and therefore shuts down the GameServer.

If you run kubectl describe gameserver again - either the GameServer will be gone completely, or it will be in Shutdown state, on the way to being deleted.

Next Steps

Make a local copy of the simple-game-server fleet configuration , modify it to include a node selector, and use it to go through the Quickstart: Create a Game Server Fleet.
If you want to use your own GameServer container make sure you have properly integrated the Agones SDK.

Docs: Fleet Updates

Tue, 27 Aug 2019 03:58:19 +0000

Rolling Update Strategy

When Fleets are edited and updated, the default strategy of Agones is to roll the new version of the GameServer out to the entire Fleet, in a step by step increment and decrement by adding a chunk of the new version and removing a chunk of the current set of GameServers.

This is done while ensuring that Allocated GameServers are not deleted until they are specifically shutdown through the game servers SDK, as they are expected to have players on them.

You can see this in the Fleet.Spec.Strategy reference, with controls for how much of the Fleet is incremented and decremented at one time:

  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 25%
      maxUnavailable: 25%

So when a Fleet is edited (any field other than replicas, see note below), either through kubectl edit/apply or via the Kubernetes API, this performs the following operations:

Adds the maxSurge number of GameServers to the Fleet.
Shutdown the maxUnavailable number of GameServers in the Fleet, skipping Allocated GameServers.
Repeat above steps until all the previous GameServer configurations have been Shutdown and deleted.

By default, a Fleet will wait for new GameSevers to become Ready during a Rolling Update before continuing to shutdown additional GameServers, only counting GameServers that are Ready as being available when calculating the current maxUnavailable value which controls the rate at which GameServers are updated. This ensures that a Fleet cannot accidentally have zero GameServers in the Ready state if something goes wrong during a Rolling Update or if GameServers have a long delay before moving to the Ready state.

Note

When Fleet update contains only changes to the replicas parameter, then new GameServers will be created/deleted straight away, which means in that case maxSurge and maxUnavailable parameters for a RollingUpdate will not be used. The RollingUpdate strategy takes place when you update spec parameters other than replicas.

If you are using a Fleet which is scaled by a FleetAutoscaler, read the Fleetautoscaler guide for more details on how RollingUpdates with FleetAutoscalers need to be implemented.

You could also check the behaviour of the Fleet with a RollingUpdate strategy on a test Fleet to preview your upcoming updates. Use kubectl describe fleet to track scaling events in a Fleet.

Recreate Strategy

This is an optimal Fleet update strategy if you want to replace all GameServers that are not Allocated with a new version as quickly as possible.

You can see this in the Fleet.Spec.Strategy reference:

  strategy:
    type: Recreate

So when a Fleet is edited, either through kubectl edit/apply or via the Kubernetes API, this performs the following operations:

Shutdown all GameServers in the Fleet that are not currently Allocated.
Create the same number of the new version of the GameServers that were previously deleted.
Repeat above steps until all the previous GameServer configurations have been Shutdown and deleted.

Two (or more) Fleets Strategy

If you want very fine-grained control over the rate that new versions of a GameServer configuration is rolled out, or if you want to do some version of A/B testing or smoke test between different versions, running two (or more) Fleets at the same time is a good solution for this.

To do this, create a second Fleet inside your cluster, starting with zero replicas. From there you can scale this newer Fleet up and the older Fleet down as required by your specific rollout strategy.

This also allows you to rollback if issues arise with the newer version, as you can delete the newer Fleet and scale up the old Fleet to its previous levels, resulting in minimal impact to the players.

Note

For GameServerAllocation, you will need to have at least a single shared label between the GameServers in each Fleet.

GameServerAllocation Across Fleets

Since GameServerAllocation is powered by label selectors, it is possible to allocate across multiple fleets, and/or give preference to particular sets of GameServers over others. You can see details of this in the GameServerAllocation reference.

In a scenario where a new v2 version of a Fleet is being slowly scaled up in a separate Fleet from the previous v1 Fleet, we can specify that we prefer allocation to occur from the v2 Fleet, and if none are available, fallback to the v1 Fleet, like so:

apiVersion: "allocation.agones.dev/v1"
kind: GameServerAllocation
spec:
  selectors:
    - matchLabels:
        agones.dev/fleet: v2
    - matchLabels:
        game: my-awesome-game

apiVersion: "allocation.agones.dev/v1"
kind: GameServerAllocation
spec:
  # Deprecated, use field selectors instead.
  required:
    matchLabels:
      game: my-awesome-game
  # Deprecated, use field selectors instead.
  preferred:
    - matchLabels:
        agones.dev/fleet: v2

In this example, all GameServers have the label game: my-awesome-game, so the Allocation will search across both Fleets through that mechanism. The selectors label matching selector tells the allocation system to first search all GameServers with the v2 Fleet label, and if not found, search through the rest of the set.

The above GameServerAllocation can then be used while you scale up the v2 Fleet and scale down the original Fleet at the rate that you deem fit for your specific rollout.

Notifying GameServers on Fleet Update/Downscale

Warning

The Allocated GameSever Overflow Notification feature is currently Beta, and while it is enabled by default it may change in the future.

Use the Feature Gate FleetAllocationOverflow to disable this feature.

See the Feature Gate documentation for details on how to disable features.

When Allocated GameServers are utilised for a long time, such as a Lobby GameServer, or a GameServer that is being reused multiple times in a row, it can be useful to notify an Allocated GameServer process when its backing Fleet has been updated. When an update occurs, the Allocated GameServer, may want to actively perform a graceful shutdown and release its resources such that it can be replaced by a new version, or similar actions.

To do this, we provide the ability to apply a user-provided set of labels and/or annotations to the Allocated GameServers when a Fleet update occurs that updates its GameServer template, or generally causes the Fleet replica count to drop below the number of currently Allocated GameServers.

This provides two useful capabilities:

The GameServer SDK.WatchGameServer() command can be utilised to react to this annotation and/or label change to indicate the Fleet system change, and the game server binary could execute code accordingly.
This can also be used to proactively update GameServer labels, to effect change in allocation strategy - such as preferring the newer GameServers when allocating, but falling back to the older version if there aren’t enough of the new ones yet spun up.

The labels and/or annotations are applied to GameServers in a Fleet in the order designated by their configured Fleet scheduling.

Example yaml configuration:

apiVersion: "agones.dev/v1"
kind: Fleet
metadata:
  name: simple-game-server
spec:
  replicas: 2
  allocationOverflow: # This specifies which annotations and/or labels are applied
    labels:
      mykey: myvalue
      version: "" # empty an existing label value, so it's no longer in the allocation selection
    annotations:
      event: overflow
  template:
    spec:
      ports:
        - name: default
          containerPort: 7654
      template:
        spec:
          containers:
            - name: simple-game-server
              image: us-docker.pkg.dev/agones-images/examples/simple-game-server:0.27

See the Fleet reference for more details.

Note

This works the same across Fleet resizing and Rolling/Recreate Updates, in that the implementation responds to the underlying GameServerSet’s replicas being shrunk to a value smaller than the number of Allocated GameServers it controls. Therefore, this functionality works equally well with a rolling update as it does with an update strategy that requires at least two Fleets.

Docs: GameServer Health Checking

Thu, 03 Jan 2019 01:20:19 +0000

Disabling Health Checking

By default, health checking is enabled, but it can be turned off by setting the spec.health.disabled property to true.

SDK API

The Health() function on the SDK object needs to be called at an interval less than the spec.health.periodSeconds threshold time to be considered before it will be considered a failure.

The health check will also need to have not been called a consecutive number of times (spec.health.failureThreshold), giving it a chance to heal if it there is an issue.

Health Failure Strategy

The following is the process for what happens to a GameServer when it is unhealthy.

If the GameServer container exits with an error before the GameServer moves to Ready then, it is restarted as per the restartPolicy (which defaults to “Always”).
If the GameServer fails health checking at any point, then it doesn’t restart, but moves to an Unhealthy state.
If the GameServer container exits while in Ready, Allocated or Reserved state, it will be restarted as per the restartPolicy (which defaults to “Always”, since RestartPolicy is a Pod wide setting), but will immediately move to an Unhealthy state.
If the SDK sidecar fails, then it will be restarted, assuming the RestartPolicy is Always/OnFailure.

Fleet Management of Unhealthy GameServers

If a GameServer moves into an Unhealthy state when it is not part of a Fleet, the GameServer will remain in the Unhealthy state until explicitly deleted. This is useful for debugging Unhealthy GameServers, or if you are creating your own GameServer management layer, you can explicitly choose what to do if a GameServer becomes Unhealthy.

If a GameServer is part of a Fleet, the Fleet management system will delete any Unhealthy GameServers and immediately replace them with a brand new GameServer to ensure it has the configured number of Replicas.

Configuration Reference

  # Health checking for the running game server
  health:
    # Disable health checking. defaults to false, but can be set to true
    disabled: false
    # Number of seconds after the container has started before health check is initiated. Defaults to 5 seconds
    initialDelaySeconds: 5
    # If the `Health()` function doesn't get called at least once every period (seconds), then
    # the game server is not healthy. Defaults to 5
    periodSeconds: 5
    # Minimum consecutive failures for the health probe to be considered failed after having succeeded.
    # Defaults to 3. Minimum value is 1
    failureThreshold: 3

See the full GameServer example for more details

Example

C++

For a configuration that requires a health ping every 5 seconds, the example below sends a request every 2 seconds to be sure that the GameServer is under the threshold.

void doHealth(agones::SDK *sdk) {
    while (true) {
        if (!sdk->Health()) {
            std::cout << "Health ping failed" << std::endl;
        } else {
            std::cout << "Health ping sent" << std::endl;
        }
        std::this_thread::sleep_for(std::chrono::seconds(2));
    }
}

int main() {
    agones::SDK *sdk = new agones::SDK();
    bool connected = sdk->Connect();
    if (!connected) {
        return -1;
    }
    std::thread health (doHealth, sdk);

    // ...  run the game server code

}

Full Game Server

Also look in the examples directory.

Docs: GameServer Counters and Lists

Mon, 08 Jan 2024 00:00:00 +0000

Warning

The Counters and Lists feature is currently Alpha, not enabled by default, and may change in the future.

Use the FeatureGate CountsAndLists to enable and test this feature.

See the Feature Gate documentation for details on how to enable features.

Counters and Lists is provided as a way to track arbitrary integer counter values as well as lists of values against a GameServer by a user provided key.

Combined with the ability to set and manipulate max capacity values at runtime for each counter and list, allows Agones to also provide Allocation, Fleet scheduling and Fleet autoscaling based on this functionality, such that it supports a wide variety of use cases, including, but not limited to:

Connected player listing tracking, Allocation filtering and autoscaling based on available capacity.
Multi-tenant server room counting, Allocation filtering and autoscaling based on available capacity.
Game specific GameServer weighting on Allocation.
…any other use case that requires a list of values or a counter aligned with a GameServer for tracking purposes.

Declaration

All keys for either Counters and Lists must be declared at creation time within the GameServerSpec before being utilised, and keys cannot be added or deleted from GameServers past their initial creation.

For example, if we want to use a Counter of rooms to track the number of game session rooms that exist in this GameServer instance, while also tracking a List of currently connected players, we could implement the following:

apiVersion: agones.dev/v1
kind: Fleet
metadata:
  name: simple-game-server
spec:
  replicas: 2
  template:
    spec:
      ports:
        - name: default
          containerPort: 7654
      counters:
        rooms: # room counter
          count: 0
          capacity: 10
      lists:
        players: # players list
          values: []
      template:
        spec:
          containers:
            - name: simple-game-server
              image: us-docker.pkg.dev/agones-images/examples/simple-game-server:0.31

Both Counters and Lists can have a capacity value, which indicated the maximum counter value, or number of items that can be stored in the list.

In the above example, the room Counter has a capacity of 10, which means that the count cannot go past that value.

See the GameServer reference for all configurable options.

Retrieval

We now have several ways to retrieve this Counter and List information from a GameServer, depending on your use case.

Kubernetes API

If you wish to retrieve or view current Counter or List values from outside your GameServer, you are able to do this through the Kubernetes API, or similarly through kubectl.

Counter values and capacities for are stored on a GameServer resource instance under GameServer.Status.Counters by key, and GameServer.Status.Lists stores List value arrays and capacities by key as well.

Therefore, in the above examples, the GameServer.Status.Counters[rooms].Count and GameServer.Status.Counters[rooms].Capacity would have the current Counter value and capacity for the room counters.

Subsequently GameServer.Status.Lists[players].Values stores the array of values for the list and GameServer.Status.Lists[players].Capacity is the current capacity for the player tracking List.

Check the API reference for GameServerStatus for all the details on the data structure.

SDK

Counter and Lists values can be accessed through the Agones SDK when that information is required within your game server process.

For example, to retrieve the above room counter, each language SDK has some implementation of the SDK.Alpha().GetCounterCount("room") function that returns the current value of the counter. Similarly, to retrieve the players list, we can use the SDK.Alpha().GetListValues("players") function.

The special ability of the SDK retrieval operations, is that is also tracks any modifications that have been made through the SDK, that have yet to be persisted to the GameServer, and will return values with that data included.

This means that any modifications made to Counters and Lists sent through the SDK can be immediately and atomically retrieved from the SDK from the game server binary with accurate information without having to wait for it be persisted to the GameServer.

See the SDK Guide for the full set of data retrieval functions that are available.

Manipulation

We also have several ways to manipulate the Counter and List information stored on a GameServer instance, depending on your use case.

SDK

Counter and Lists values can be modified through the Agones SDK when you wish to be able to edit that information within your game server process.

For example, to increment the above room counter by 1, each language SDK has some implementation of the SDK.Alpha().IncrementCounter("room", 1) function thatincrements the counter by 1. Similarly, to add the value player1 to the players list, we can use the SDK.Alpha().AppendListValue("players", "player1") function.

See the SDK Guide for the full set of modification functions that are available.

Allocation

When performing a GameServer allocation, you may want to manipulate the Counter and/or List information atomically on a GameServer allocation operation.

For example, you may wish to increment the room counter to indicate a new game session has started on the returned GameServer instance, or provide the connecting player information to the player list to make that available to the game server binary before the player connects.

This can be done through GameServerAllocation.Spec.Counters and GameServerAllocation.Spec.Lists, which provide CounterAction and ListAction configuration respectively.

For example, if on allocation we wished to increment the room counter by 1 and add player1 to the players list, this could be done with the following GameServerAllocation specification:

apiVersion: allocation.agones.dev/v1
kind: GameServerAllocation
spec:
  selectors:
    - matchLabels:
        agones.dev/fleet: simple-game-server
  counters:
    rooms:
      action: Increment
      amount: 1
  lists:
    players:
      addValues:
        - player1

Counter and List changes made through the GameServerAllocation functionality, will be eventually consistent in their availability via the SDK retrieval functions for performance reasons.

Performing these data changes as part of the Allocation also does not guarantee that the counter or list is past its capacity value. In the event that the capacity is exceeded, the allocation will succeed, but the Counter or List values will not be updated. If you want to ensure there is capacity available for the Allocation operation, see Allocation Filtering and Prioritisation below.

See the Allocation Reference for all the Allocation configuration options.

Kubernetes API

Counter values and capacities are stored on a GameServer resource instance under GameServer.Status.Counters by key, and GameServer.Status.Lists stores List value arrays and capacities by key as well. Therefore, they can be modified through the Kubernetes API either through Kubernetes client libraries or manually through kubectl edit.

Counter and List changes made through the GameServerAllocation functionality, will be eventually consistent in their availability via the SDK retrieval functions for performance reasons.

Check out the Access Agones via the Kubernetes API guide for a more in depth guide on how to interact with Agones resources through the Kubernetes API programmatically.

Allocation Filtering and Prioritisation

Counters and Lists can also be used as filtering properties when performing an allocation through either the GameServerAllocation.Spec.Selectors.Counts or GameServerAllocation.Spec.Selectors.Lists properties.

If we want to expand the above example to ensure there is always room for the room Counter increment and room for the player List to add player1, we can add the following to the GameServerAllocation specification:

apiVersion: allocation.agones.dev/v1
kind: GameServerAllocation
spec:
  selectors:
    - matchLabels:
        agones.dev/fleet: simple-game-server
      counters:
        rooms:
          minAvailable: 1
      lists:
        players:
          minAvailable: 1
  counters:
    rooms:
      action: Increment
      amount: 1
  lists:
    players:
      addValues:
        - player1

Only GameServer instances that have a capacity value of at least one more than the Counter value, or List length will be included in the potential allocation result - thereby ensuring that both room and player operations will succeed, assuming a GameServer is found.

This can be combined with the ability to define multiple selectors to create quite sophisticated GameServer selection options that can attempt to find the most appropriate GameServer.

See the Allocation Reference for more details.

The priorities block also gives you options to prioritise specific GameServers over others when performing an Allocation, depending on the Allocation Strategy.

Packed: The usual infrastructure optimisation strategy still applies, but the priorities block is used as a tie-breaker within the least utilised infrastructure, to ensure optimal infrastructure usage while also allowing some custom prioritisation of GameServers.
Distributed: The entire selection of GameServers will be sorted by this priority list to provide the order that GameServers will be allocated by.

For example, if we wanted to select a GameServer that had:

At least one room available in the GameServer capacity.
First check Allocated GameServers and falling back to Ready GameServers if there aren’t any.
Include general infrastructure optimisation for least usage.
And also ensure the GameServers that were most full are allocated first (after infrastructure optimisations where applied).

We could implement the following GameServerAllocation:

apiVersion: allocation.agones.dev/v1
kind: GameServerAllocation
spec:
  scheduling: Packed
  # Choose most full `GameServers` first
  priorities:
    - type: Counter
      key: rooms
      order: Ascending
  selectors:
    # First check to see if we can back-fill an already allocated `GameServer`
    - gameServerState: Allocated
      matchLabels:
        agones.dev/fleet: simple-game-server
      counters:
        rooms:
          minAvailable: 1
    # If we can't, then go get a `Ready` `GameServer`.
    - gameServerState: Ready
      matchLabels:
        agones.dev/fleet: simple-game-server
      counters:
        rooms:
          minAvailable: 1
  counters:
    rooms:
      action: Increment
      amount: 1

For more details on how Agones implements infrastructure optimisation, see the documentation on Scheduling and Autoscaling.

Fleet Scale Down Prioritisation

Another optimisation control you can apply with Fleets when Agones’ scaled them down, is to also set priorities to influence the order in which GameServers are shutdown and deleted.

Warning

In versions 1.39.0 and lower, Fleet prioritisation is calculated using total capacity, not available capacity. This is being updated in the 1.40.0 release such that this and all uses of priorities operate using available capacity, and unless otherwise noted, will be the default behaviour in future releases.

Therefore, in this rare case, the documentation on this functionality is forward-looking.

While neither players or rooms are particularly good examples for this functionality, if we wanted to ensure that Ready GameServers with the most available capacity rooms where a factor when scaling down a Fleet we could implement the following:

apiVersion: agones.dev/v1
kind: Fleet
metadata:
  name: simple-game-server
spec:
  replicas: 2
  priorities:
    - type: Counter
      key: rooms
      order: Descending
  template:
    spec:
      ports:
        - name: default
          containerPort: 7654
      counters:
        rooms: # room counter
          count: 0
          capacity: 10
      template:
        spec:
          containers:
            - name: simple-game-server
              image: us-docker.pkg.dev/agones-images/examples/simple-game-server:0.31

Depending on the Fleet allocation strategy, the priorities block will influence scale down logic as follows:

Packed: The usual infrastructure optimisation strategy still applies, but the priorities block is used as a tie-breaker within the least utilised infrastructure, to ensure optimal infrastructure usage while also allowing some custom prioritisation of GameServers.
Distributed: The entire selection of GameServers will be sorted by this priority list to provide the order that GameServers will be scaled down by.

See Fleet Reference for all the configuration options.

Autoscaling

Counters and Lists expands on Fleet Autoscaling capabilities, by allowing you to autoscale based on available capacity across the Fleet as a unit, rather than the less granular unit of individual GameServers instances.

This means, we can implement an Autoscaling strategy of “always make sure there are 5 free rooms available for new players at all times” like so:

apiVersion: autoscaling.agones.dev/v1
kind: FleetAutoscaler
metadata:
  name: simple-game-server
spec:
  fleetName: fleet-example
  policy:
    type: Counter
    counter:
      key: rooms
      bufferSize: 5
      maxCapacity: 100

See the Fleet Autoscaling Reference for all the configuration options.

Metrics

Metrics are exported, using the key that the metric is stored under as a label on the metrics, in aggregate across all GameServers within a Fleet, exporting aggregate numeric totals for Counters and Lists as gauge metrics.

Name	Description	Type
agones_fleet_counters	Aggregate Metrics for Counters within a Fleet, including total capacity and count values	gauge
agones_fleet_lists	Aggregate Metrics for Lists within a Fleet, including total capacity and List lengths	gauge

See Metrics available for the full list of available metrics and how to use them.

Next Steps

Check out how to access Agones resource via the Kubernetes API.
Have a look at the external Allocator Service to make integrating Allocation into your workflow easier.

Docs: Player Tracking

Tue, 19 May 2020 00:00:00 +0000

Counters and Lists will eventually replace the Alpha functionality of Player Tracking, which will subsequently be removed from Agones.

If you are currently using this Alpha feature, we would love for you to test (and ideally migrate to!) this new functionality to Counters and Lists to ensure it meet all your needs.

Warning

The Player Tracking feature is currently Alpha, not enabled by default, and may change in the future.

Use the FeatureGate PlayerTracking to enable and test this feature.

See the Feature Gate documentation for details on how to enable features.

Managing GameServer Capacities

To track your GameServer current player capacity, Agones gives you the ability to both set an initial capacity at GameServer creation, as well be able to change it during the lifecycle of the GameServer through the Agones SDK.

To set the initial capacity, you can do so via GameServer.Spec.Players.InitialCapacity like so:

apiVersion: "agones.dev/v1"
kind: GameServer
metadata:
  name: "gs-example"
spec:
  # ...
  players:
    # set this GameServer's initial player capacity to 10
    initialCapacity: 10

From there, if you need to change the capacity of the GameSever as gameplay is in progress, you can also do so via SDK.Alpha().SetPlayerCapacity(count)

The current player capacity is represented in GameServer.Status.Players.Capacity resource value.

We can see this in action, when we look at the Status section of a GameServer resource , wherein the capacity has been set to 20:

...
Status:
  Address:    14.81.195.72
  Node Name:  gke-test-cluster-default-6cd0ba67-1mps
  Players:
    Capacity:  20
    Count:     0
    Ids:       <nil>
  Ports:
    Name:          gameport
    Port:          7983
  Reserved Until:  <nil>
  State:           Ready

From the SDK, the game server binary can also retrieve the current player capacity via SDK.Alpha().GetPlayerCapacity().

Note

Changing the capacity value here has no impact on players actually connected to or trying to connect to your server, as that is not a responsibility of Agones.

This functionality is for tracking purposes only.

Connecting and Disconnecting Players

As players connect and disconnect from your game, the Player Tracking functions enable you to track which players are currently connected.

It assumed that each player that connects has a unique token that identifies them as a player.

When a player connects to the game server binary, calling SDK.Alpha().PlayerConnect(playerID) with the unique player token will register them as connected, and store their player id.

At disconnection time, call SDK.Alpha().PlayerDisconnect(playerID) , which will deregister them and remove their player id from the list.

Each of these playerIDs is stored on GameServer.Status.Players.IDs, and the current count of connected players can be seen in GameServer.Status.Players.Count.

You can see this in action below in the GameServer Status section, where there are 4 players connected:

...
Status:
  Address:    39.82.196.74
  Node Name:  gke-test-cluster-default-6cd0ba77-1mps
  Players:
    Capacity:  10
    Count:     4
    Ids:
      xy8a
      m0ux
      71nj
      lpq5
  Ports:
    Name:          gameport
    Port:          7166
  Reserved Until:  <nil>
  State:           Ready

Note

Calling PlayerConnect or PlayerDisconnect functions will not connect or disconnect players, as that is not under the control of Agones.

This functionality is for tracking purposes only.

Checking Player Data

Not only is the connected player data stored on the GameServer resource, it is also stored in memory within the SDK, so that it can be used from within the game server binary as a realtime, thread safe, registry of connected players.

Therefore, if you want to:

Get the current player count, call SDK.Alpha().GetPlayerCount()
Check if a specific player is connected, call SDK.Alpha().IsPlayerConnected(playerID)
Retrieve the full list of connected players, call SDK.Alpha().GetConnectedPlayers()

Next Steps

Review the Player Tracking SDK Reference

Docs: Local Game Server

Tue, 19 Feb 2019 08:00:00 +0000

You can register a local game server with Agones. This means you can run an experimental build of your game server in the Agones environment without the need of packaging and deploying it to a fleet. This allows you to quickly iterate on your game server code while still being able to plugin to your Agones environment.

Register your server with Agones

To register your local game server you’ll need to know the IP address of the machine running it and the port. With that you’ll create a game server config like the one below.

apiVersion: "agones.dev/v1"
kind: GameServer
metadata:
  name: my-local-server
  annotations:
    # Causes Agones to register your local game server at 192.1.1.2, replace with your server's IP address.
    agones.dev/dev-address: "192.1.1.2"
spec:
  ports:
  - name: default
    portPolicy: Static
    hostPort: 17654
    containerPort: 17654
  # The following is ignored but required due to validation.
  template:
    spec:
      containers:
      - name: simple-game-server
        image: us-docker.pkg.dev/agones-images/examples/simple-game-server:0.31

Once you save this to a file make sure you have kubectl configured to point to your Agones cluster and then run kubectl apply -f dev-gameserver.yaml. This will register your server with Agones.

Local Game Servers has a few limitations:

PortPolicy must be Static.
The game server is not managed by Agones. Features like autoscaling, replication, etc are not available.

When you are finished working with your server, you can remove the registration with kubectl delete -f dev-gameserver.yaml

Next Steps:

Review the specification of GameServer.
Read about GameServer allocation.
- Review the flow of how allocation is done.
- Review the specification of GameServerAllocation.
- Check out the Allocator Service as a richer alternative to GameServerAllocation.
Learn how to connect your local development game server binary into a running Agones Kubernetes cluster for even more live development options with an out of cluster dev server.

Docs: Latency Testing with Multiple Clusters

Thu, 03 Jan 2019 01:20:30 +0000

To make latency testing easier, Agones installs with a simple ping service with both HTTP and UDP services that can be called for the purpose of timing how long the roundtrip takes for information to be returned from either of these services.

Installing

By default, Agones installs Kubernetes Services for both HTTP and the UDP ping endpoints. These can be disabled entirely, or disabled individually. See the Helm install guide for the parameters to pass through, as well as configuration options.

The ping services as all installed under the agones-system namespace.

HTTP Service

This exposes an endpoint that returns a simple text HTTP response on request to the root “/” path. By default this is ok, but it can be configured via the agones.ping.http.response parameter.

This could be useful for providing clusters with unique lookup names, such that clients are able to identify clusters from their responses.

To lookup the details of this service, run kubectl describe service agones-ping-http-service --namespace=agones-system

UDP Service

The UDP ping service is a rate limited UDP echo service that returns the udp packet that it receives to its designated sender.

Since UDP sender details can be spoofed, this service is rate limited to 20 requests per second, per sender address, per running instance (default is 2).

This rate limit can be raised or lowered via the Helm install parameter agones.ping.udp.rateLimit.

UDP packets are also limited to 1024 bytes in size.

To lookup the details of this service, run kubectl describe service agones-ping-udp-service --namespace=agones-system

Client side tooling

We deliberately didn’t provide any game client libraries, as all major languages and engines have capabilities to send HTTP requests as well as UDP packets.

Docs: Metrics

Thu, 03 Jan 2019 03:58:19 +0000

Agones controller exposes metrics via OpenCensus. OpenCensus is a single distribution of libraries that collect metrics and distributed traces from your services, we only use it for metrics but it will allow us to support multiple exporters in the future.

We choose to start with Prometheus as this is the most popular with Kubernetes but it is also compatible with Cloud Monitoring. If you need another exporter, check the list of supported exporters. It should be pretty straightforward to register a new one. (GitHub PRs are more than welcome.)

We plan to support multiple exporters in the future via environment variables and helm flags.

Backend integrations

Prometheus

If you are running a Prometheus instance you just need to ensure that metrics and kubernetes service discovery are enabled. (helm chart values agones.metrics.prometheusEnabled and agones.metrics.prometheusServiceDiscovery). This will automatically add annotations required by Prometheus to discover Agones metrics and start collecting them. (see example)

If your Prometheus metrics collection agent requires that you scrape from the pods directly(such as with Google Cloud Managed Prometheus), then the metrics ports for the controller and allocator will both be named http and exposed on 8080. In the case of the allocator, the port name and number can be overriden with the agones.allocator.serviceMetrics.http.portName and agones.allocator.serviceMetrics.http.port helm chart values.

Prometheus Operator

If you have Prometheus operator installed in your cluster, just enable ServiceMonitor installation in values:

agones:
  metrics:
    serviceMonitor:
      enabled: true

Google Cloud Managed Service for Prometheus

Google Cloud Managed Service for Prometheus is a fully managed multi-cloud solution for Prometheus. If you wish to use Managed Prometheus with Agones, follow the Google Cloud Managed Service for Prometheus installation steps.

Google Cloud Monitoring (formerly Stackdriver)

We support the OpenCensus Stackdriver exporter. In order to use it you should enable Cloud Monitoring API in Google Cloud Console. Follow the Google Cloud Monitoring installation steps to see your metrics in Cloud Monitoring.

Metrics available

Name	Description	Type
agones_gameservers_count	The number of gameservers per fleet and status	gauge
agones_gameserver_allocations_duration_seconds	The distribution of gameserver allocation requests latencies	histogram
agones_gameservers_total	The total of gameservers per fleet and status	counter
agones_gameserver_player_connected_total	The total number of players connected to gameservers (Only available when player tracking is enabled)	gauge
agones_gameserver_player_capacity_total	The available capacity for players on gameservers (Only available when player tracking is enabled)	gauge
agones_fleets_replicas_count	The number of replicas per fleet (total, desired, ready, reserved, allocated)	gauge
agones_fleet_counters	Aggregate Metrics for Counters within a Fleet, including total capacity and count values (Only available when Counters and Lists) are enabled)	gauge
agones_fleet_lists	Aggregate Metrics for Lists within a Fleet, including total capacity and List lengths (Only available when Counters and Lists) are enabled)	gauge
agones_fleet_autoscalers_able_to_scale	The fleet autoscaler can access the fleet to scale	gauge
agones_fleet_autoscalers_buffer_limits	The limits of buffer based fleet autoscalers (min, max)	gauge
agones_fleet_autoscalers_buffer_size	The buffer size of fleet autoscalers (count or percentage)	gauge
agones_fleet_autoscalers_current_replicas_count	The current replicas count as seen by autoscalers	gauge
agones_fleet_autoscalers_desired_replicas_count	The desired replicas count as seen by autoscalers	gauge
agones_fleet_autoscalers_limited	The fleet autoscaler is outside the limits set by MinReplicas and MaxReplicas.	gauge
agones_gameservers_node_count	The distribution of gameservers per node	histogram
agones_nodes_count	The count of nodes empty and with gameservers	gauge
agones_gameservers_state_duration	The distribution of gameserver state duration in seconds. Note: this metric could have some missing samples by design. Do not use the `_total` counter as the real value for state changes.	histogram
agones_k8s_client_http_request_total	The total of HTTP requests to the Kubernetes API by status code	counter
agones_k8s_client_http_request_duration_seconds	The distribution of HTTP requests latencies to the Kubernetes API by status code	histogram
agones_k8s_client_cache_list_total	The total number of list operations for client-go caches	counter
agones_k8s_client_cache_list_duration_seconds	Duration of a Kubernetes list API call in seconds	histogram
agones_k8s_client_cache_list_items	Count of items in a list from the Kubernetes API	histogram
agones_k8s_client_cache_watches_total	The total number of watch operations for client-go caches	counter
agones_k8s_client_cache_last_resource_version	Last resource version from the Kubernetes API	gauge
agones_k8s_client_workqueue_depth	Current depth of the work queue	gauge
agones_k8s_client_workqueue_latency_seconds	How long an item stays in the work queue	histogram
agones_k8s_client_workqueue_items_total	Total number of items added to the work queue	counter
agones_k8s_client_workqueue_work_duration_seconds	How long processing an item from the work queue takes	histogram
agones_k8s_client_workqueue_retries_total	Total number of items retried to the work queue	counter
agones_k8s_client_workqueue_longest_running_processor	How long the longest running workqueue processor has been running in microseconds	gauge
agones_k8s_client_workqueue_unfinished_work_seconds	How long unfinished work has been sitting in the workqueue in seconds	gauge

Dropping Metric Labels

When a Fleet or FleetAutoscaler is deleted from the system, Agones will automatically clear metrics that utilise their name as a label from the exported metrics, so the metrics exported do not continuously grow in size over the lifecycle of the Agones installation.

Dashboard

Grafana Dashboards

We provide a set of useful Grafana dashboards to monitor Agones workload, they are located under the grafana folder :

Agones Autoscalers allows you to monitor your current autoscalers replicas request as well as fleet replicas allocation and readyness statuses. You can only select one autoscaler at the time using the provided dropdown.
Agones GameServers displays your current game servers workload status (allocations, game servers statuses, fleets replicas) with optional fleet name filtering.
Agones GameServer Allocations displays Agones gameservers allocations rates and counts per fleet.
Agones Allocator Resource displays Agones Allocators CPU, memory usage and also some useful Golang runtime metrics.
Agones Status displays Agones controller health status.
Agones Controller Resource Usage displays Agones Controller CPU and memory usage and also some Golang runtime metrics.
Agones Controller go-client requests displays Agones Controller Kubernetes API consumption.
Agones Controller go-client caches displays Agones Controller Kubernetes Watches/Lists operations used.
Agones Controller go-client workqueues displays Agones Controller workqueue processing time and rates.
Agones Controller API Server requests displays your current API server request rate, errors rate and request latencies with optional CustomResourceDefinition filtering by Types: fleets, gameserversets, gameservers, gameserverallocations.

Dashboard screenshots :

Note

You can import our dashboards by copying the json content from each config map into your own instance of Grafana (+ > Create > Import > Or paste json) or follow the installation guide.

Installation

When operating a live multiplayer game you will need to observe performances, resource usage and availability to learn more about your system. This guide will explain how you can setup Prometheus and Grafana into your own Kubernetes cluster to monitor your Agones workload.

Before attemping this guide you should make sure you have kubectl and helm installed and configured to reach your kubernetes cluster.

Prometheus installation

Prometheus is an open source monitoring solution, we will use it to store Agones controller metrics and query back the data.

Let’s install Prometheus using the Prometheus Community Kubernetes Helm Charts repository.

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update

helm upgrade --install --wait prom prometheus-community/prometheus --namespace metrics --create-namespace \
    --set server.global.scrape_interval=30s \
    --set server.persistentVolume.enabled=true \
    --set server.persistentVolume.size=64Gi \
    -f ./build/prometheus.yaml

For resiliency it is recommended to run Prometheus on a dedicated node which is separate from nodes where Game Servers are scheduled. If you use the above command, with our prometheus.yaml to set up Prometheus, it will schedule Prometheus pods on nodes tainted with agones.dev/agones-metrics=true:NoExecute and labeled with agones.dev/agones-metrics=true if available.

As an example, to set up a dedicated node pool for Prometheus on GKE, run the following command before installing Prometheus. Alternatively you can taint and label nodes manually.

gcloud container node-pools create agones-metrics --cluster=... --zone=... \
  --node-taints agones.dev/agones-metrics=true:NoExecute \
  --node-labels agones.dev/agones-metrics=true \
  --num-nodes=1 \
  --machine-type=e2-standard-4

By default we will disable the push gateway (we don’t need it for Agones) and other exporters.

The helm chart supports nodeSelector, affinity and toleration, you can use them to schedule Prometheus deployments on an isolated node(s) to have an homogeneous game servers workload.

This will install a Prometheus Server in your current cluster with Persistent Volume Claim (Deactivated for Minikube and Kind) for storing and querying time series, it will automatically start collecting metrics from Agones Controller.

Finally, to access Prometheus metrics, rules and alerts explorer use

kubectl port-forward deployments/prom-prometheus-server 9090 -n metrics

Now you can access the prometheus dashboard http://localhost:9090.

On the landing page you can start exploring metrics by creating queries. You can also verify what targets Prometheus currently monitors (Header Status > Targets), you should see Agones controller pod in the kubernetes-pods section.

Note

Metrics will be first registered when you will start using Agones.

Now let’s install some Grafana dashboards.

Grafana installation

Grafana is a open source time series analytics platform which supports Prometheus data source. We can also easily import pre-built dashboards.

First we will install Agones dashboard as config maps in our cluster.

kubectl apply -f ./build/grafana/

Now we can install the Grafana Community Kubernetes Helm Charts from their repository. (Replace <your-admin-password> with the admin password of your choice)

helm repo add grafana https://grafana.github.io/helm-charts
helm repo update

helm upgrade --install --wait grafana grafana/grafana --namespace metrics \
  --set adminPassword=<your-admin-password> -f ./build/grafana.yaml

This will install Grafana with our prepopulated dashboards and prometheus datasource previously installed

Finally to access dashboards run

kubectl port-forward deployments/grafana 3000 -n metrics

Open a web browser to http://localhost:3000, you should see Agones dashboards after login as admin.

Google Cloud Managed Service for Prometheus installation

To collect Agones metrics using Managed Prometheus:

Follow the instructions to enable managed collection for a GKE cluster or non-GKE cluster.
Configure Managed Prometheus to scrape Agones by creating a PodMonitoring resource:

kubectl apply -n agones-system -f https://raw.githubusercontent.com/googleforgames/agones/release-1.39.0/build/prometheus-google-managed.yaml

Confirm that you can see Prometheus metrics in Cloud Monitoring. If that’s all you need, you can stop here.

To install Grafana using a Managed Prometheus backend:

Complete the Before you begin. To align with the Agones Grafana installation, we’ll be installing in the metrics namespace, which you’ll need to create.
- If your cluster has Workload Identity enabled, which is enabled on GKE Autopilot by default, follow Configure a service account for Workload Identity to ensure that you have appropriately authorized the default Kubernetes service account in the metrics namespace.
Install the Standalone Prometheus frontend UI in the metrics namespace - this will act as your authentication proxy for PromQL queries.
Install Grafana as above, using -f ./build/grafana-frontend.yaml instead of -f ./build/grafana.yaml.

Google Cloud Monitoring installation

In order to use Google Cloud Monitoring you must enable the Monitoring API in the Google Cloud Console. The Cloud Monitoring exporter uses a strategy called Application Default Credentials (ADC) to find your application’s credentials. Details can be found in Setting Up Authentication for Server to Server Production Applications.

You need to grant all the necessary permissions to the users (see Access Control Guide). The predefined role Monitoring Metric Writer contains those permissions. Use the following command to assign the role to your default service account.

gcloud projects add-iam-policy-binding [PROJECT_ID] --member serviceAccount:[PROJECT_NUMBER]-compute@developer.gserviceaccount.com --role roles/monitoring.metricWriter

Note

Cloud Operations for GKE (including Cloud Monitoring) is enabled by default on GKE clusters, however you can follow this guide if it is currently disabled in your GKE cluster.

Before proceeding, ensure you have created a metrics node pool as mentioned in the Google Cloud installation guide.

The default metrics exporter installed with Agones is Prometheus. If you are using the Helm installation, you can install or upgrade Agones to use Cloud Monitoring, using the following chart parameters:

helm upgrade --install --wait --set agones.metrics.stackdriverEnabled=true --set agones.metrics.prometheusEnabled=false --set agones.metrics.prometheusServiceDiscovery=false my-release-name agones/agones --namespace=agones-system

Note

If you are using the YAML installation, follow the instructions on the page to change the above parameters by using helm to generate a custom YAML file locally.

With this configuration only the Cloud Monitoring exporter would be used instead of Prometheus exporter.

Using Cloud Monitoring with Workload Identity

If you would like to enable Cloud Monitoring in conjunction with Workload Identity, there are a few extra steps you need to follow:

When setting up the Google service account following the instructions for Authenticating to Google Cloud, create two IAM policy bindings, one for serviceAccount:PROJECT_ID.svc.id.goog[agones-system/agones-controller] and one for serviceAccount:PROJECT_ID.svc.id.goog[agones-system/agones-allocator].
Pass parameters to helm when installing Agones to add annotations to the agones-controller and agones-allocator Kubernetes service accounts:

helm install my-release --namespace agones-system --create-namespace agones/agones --set agones.metrics.stackdriverEnabled=true --set agones.metrics.prometheusEnabled=false --set agones.metrics.prometheusServiceDiscovery=false --set agones.serviceaccount.allocator.annotations."iam\.gke\.io/gcp-service-account"="GSA_NAME@PROJECT_ID\.iam\.gserviceaccount\.com" --set agones.serviceaccount.allocator.labels."iam\.gke\.io/gcp-service-account"="GSA_NAME@PROJECT_ID\.iam\.gserviceaccount\.com" --set agones.serviceaccount.controller.annotations."iam\.gke\.io/gcp-service-account"="GSA_NAME@PROJECT_ID\.iam\.gserviceaccount\.com"

To verify that metrics are being sent to Cloud Monitoring, create a Fleet or a Gameserver and look for the metrics to show up in the Cloud Monitoring dashboard. Navigate to the Metrics explorer and search for metrics with the prefix agones/. Select a metric and look for data to be plotted in the graph to the right.

An example of a custom dashboard is:

Currently there exists only manual way of configuring Cloud Monitoring Dashboard. So it is up to you to set an Alignment Period (minimal is 1 minute), GroupBy, Filter parameters and other graph settings.

Troubleshooting

If you can’t see Agones metrics you should have a look at the controller logs for connection errors. Also ensure that your cluster has the necessary credentials to interact with Cloud Monitoring. You can configure stackdriverProjectID manually, if the automatic discovery is not working.

Permissions problem example from controller logs:

Failed to export to Stackdriver: rpc error: code = PermissionDenied desc = Permission monitoring.metricDescriptors.create denied (or the resource may not exist).

If you receive this error, ensure your service account has the role or corresponding permissions mentioned above.

Docs: Access Agones via the Kubernetes API

Thu, 03 Jan 2019 01:20:41 +0000

Installing Agones creates several Custom Resource Definitions (CRD), which can be accessed and manipulated through the Kubernetes API.

The detailed list of Agones CRDs with their parameters could be found here - Agones CRD API Reference.

Kubernetes has multiple client libraries, however, at time of writing, only the Go and Python clients are documented to support accessing CRDs.

This can be found in the Accessing a custom resource section of the Kubernetes documentation.

At this time, we recommend interacting with Agones through the Go client that has been generated in this repository, but other methods may also work as well.

Go Client

Kubernetes Go Client tooling generates a Client for Agones that we can use to interact with the Agones installation on our Kubernetes cluster.

Authentication

This client uses the same authentication mechanisms as the Kubernetes API.

If you plan to run your code in the same cluster as the Agones install, have a look at the in cluster configuration example from the Kubernetes Client.

If you plan to run your code outside the Kubernetes cluster as your Agones install, look at the out of cluster configuration example from the Kubernetes client.

Example

The following is an example of a in-cluster configuration, that creates a Clientset for Agones and then creates a GameServer.

A full example code is available in the example folder .

package main

import (
	"fmt"
	"context"

	agonesv1 "agones.dev/agones/pkg/apis/agones/v1"
	"agones.dev/agones/pkg/client/clientset/versioned"
	"agones.dev/agones/pkg/util/runtime"
	corev1 "k8s.io/api/core/v1"
	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
	"k8s.io/client-go/kubernetes"
	"k8s.io/client-go/rest"
)

func main() {
	config, err := rest.InClusterConfig()
	logger := runtime.NewLoggerWithSource("main")
	if err != nil {
		logger.WithError(err).Fatal("Could not create in cluster config")
	}

	// Access to standard Kubernetes resources through the Kubernetes Clientset
	// We don't actually need this for this example, but it's just here for
	// illustrative purposes
	kubeClient, err := kubernetes.NewForConfig(config)
	if err != nil {
		logger.WithError(err).Fatal("Could not create the kubernetes clientset")
	}

	// Access to the Agones resources through the Agones Clientset
	// Note that we reuse the same config as we used for the Kubernetes Clientset
	agonesClient, err := versioned.NewForConfig(config)
	if err != nil {
		logger.WithError(err).Fatal("Could not create the agones api clientset")
	}

	// Create a GameServer
	gs := &agonesv1.GameServer{ObjectMeta: metav1.ObjectMeta{GenerateName: "simple-game-server", Namespace: "default"},
		Spec: agonesv1.GameServerSpec{
			Container: "simple-game-server",
			Ports: []agonesv1.GameServerPort{{
				ContainerPort: 7654,
				HostPort:      7654,
				Name:          "gameport",
				PortPolicy:    agonesv1.Static,
				Protocol:      corev1.ProtocolUDP,
			}},
			Template: corev1.PodTemplateSpec{
				Spec: corev1.PodSpec{
					Containers: []corev1.Container{{Name: "simple-game-server", Image: "us-docker.pkg.dev/agones-images/examples/simple-game-server:0.31"}},
				},
			},
		},
	}
	newGS, err := agonesClient.AgonesV1().GameServers("default").Create(context.TODO(), gs, metav1.CreateOptions{})
	if err != nil {
		panic(err)
	}

	fmt.Printf("New game servers' name is: %s", newGS.ObjectMeta.Name)
}

In order to create GS using provided example, you can run it as a Kubernetes Job:

kubectl create -f https://raw.githubusercontent.com/googleforgames/agones/release-1.39.0/examples/crd-client/create-gs.yaml --namespace agones-system
kubectl get pods --namespace agones-system

NAME                                 READY   STATUS      RESTARTS   AGE
create-gs-6wz86-7qsm5                0/1     Completed   0          6s

kubectl logs create-gs-6wz86-7qsm5  --namespace agones-system

{"message":"\u0026{0xc0001dde00 default}","severity":"info","source":"main","time":"2020-04-21T11:14:00.477576428Z"}
{"message":"New GameServer name is: helm-test-server-fxfgg","severity":"info","time":"2020-04-21T11:14:00.516024697Z"}

You have just created a GameServer using Kubernetes Go Client.

Best Practice: Using Informers and Listers

Almost all Kubernetes’ controllers and custom controllers utilise Informers and Listers to reduce the load on the Kubernetes’s control plane.

Repetitive, direct access of the Kubernetes control plane API can significantly reduce the performance of the cluster – and Informers and Listers help resolving that issue.

Informers and Listers reduce the load on the Kubernetes control plane by creating, using and maintaining an eventually consistent an in-memory cache. This can be watched and also queried with zero cost, since it will only read against its in-memory model of the Kubernetes resources.

Informer’s role and Lister’s role are different.

An Informer is the mechanism for watching a Kubernetes object’s event, such that when a Kubernetes object changes(e.g. CREATE,UPDATE,DELETE), the Informer is informed, and can execute a callback with the relevant object as an argument.

This can be very useful for building event based systems against the Kubernetes API.

A Lister is the mechanism for querying Kubernetes object’s against the client side in-memory cache. Since the Lister stores objects in an in-memory cache, queries against a come at practically no cost.

Of course, Agones itself also uses Informers and Listers in its codebase.

Example

The following is an example of Informers and Listers, that show the GameServer’s name & status & IPs in the Kubernetes cluster.

package main

import (
	"context"
	"time"

	"agones.dev/agones/pkg/client/clientset/versioned"
	"agones.dev/agones/pkg/client/informers/externalversions"
	"agones.dev/agones/pkg/util/runtime"
	"k8s.io/apimachinery/pkg/labels"
	"k8s.io/client-go/informers"
	"k8s.io/client-go/kubernetes"
	"k8s.io/client-go/tools/cache"
)

func main() {
	config, err := rest.InClusterConfig()
	logger := runtime.NewLoggerWithSource("main")
	if err != nil {
		logger.WithError(err).Fatal("Could not create in cluster config")
	}
	kubeClient, err := kubernetes.NewForConfig(config)
	agonesClient, err := versioned.NewForConfig(config)
	if err != nil {
		logger.WithError(err).Fatal("Could not create the agones api clientset")
	}

	// Create InformerFactory which create the informer
	informerFactory := informers.NewSharedInformerFactory(kubeClient, time.Second*30)
	agonesInformerFactory := externalversions.NewSharedInformerFactory(agonesClient, time.Second*30)

	// Create Pod informer by informerFactory
	podInformer := informerFactory.Core().V1().Pods()

	// Create GameServer informer by informerFactory
	gameServers := agonesInformerFactory.Agones().V1().GameServers()
	gsInformer := gameServers.Informer()

	// Add EventHandler to informer
	// When the object's event happens, the function will be called
	// For example, when the pod is added, 'AddFunc' will be called and put out the "Pod Added"
	podInformer.Informer().AddEventHandler(cache.ResourceEventHandlerFuncs{
		AddFunc:    func(new interface{}) { logger.Infof("Pod Added") },
		UpdateFunc: func(old, new interface{}) { logger.Infof("Pod Updated") },
		DeleteFunc: func(old interface{}) { logger.Infof("Pod Deleted") },
	})
	gsInformer.AddEventHandler(cache.ResourceEventHandlerFuncs{
		AddFunc:    func(new interface{}) { logger.Infof("GameServer Added") },
		UpdateFunc: func(old, new interface{}) { logger.Infof("GameServer Updated") },
		DeleteFunc: func(old interface{}) { logger.Infof("GameServer Deleted") },
	})

	ctx := context.Background()

	// Start Go routines for informer
	informerFactory.Start(ctx.Done())
	agonesInformerFactory.Start(ctx.Done())
	// Wait until finish caching with List API
	informerFactory.WaitForCacheSync(ctx.Done())
	agonesInformerFactory.WaitForCacheSync(ctx.Done())

	// Create Lister which can list objects from the in-memory-cache
	podLister := podInformer.Lister()
	gsLister := gameServers.Lister()

	for {
		// Get List objects of Pods from Pod Lister
		p := podLister.Pods("default")
		// Get List objects of GameServers from GameServer Lister
		gs, err := gsLister.List(labels.Everything())
		if err != nil {
			panic(err)
		}
		// Show GameServer's name & status & IPs
		for _, g := range gs {
			a, err := p.Get(g.GetName())
			if err != nil {
				panic(err)
			}
			logger.Infof("------------------------------")
			logger.Infof("Name: %s", g.GetName())
			logger.Infof("Status: %s", g.Status.State)
			logger.Infof("External IP: %s", g.Status.Address)
			logger.Infof("Internal IP: %s", a.Status.PodIP)
		}
		time.Sleep(time.Second * 25)
	}
}

You can list GameServer’s name and status and IPs using Kubernetes Informers and Listers.

Direct Access to the REST API via Kubectl

If there isn’t a client written in your preferred language, it is always possible to communicate directly with Kubernetes API to interact with Agones.

The Kubernetes API can be authenticated and exposed locally through the kubectl proxy

For example:

kubectl proxy &

Starting to serve on 127.0.0.1:8001

List all Agones endpoints

curl http://localhost:8001/apis | grep agones -A 5 -B 5

{
    "name": "agones.dev",
    "versions": [
    {
        "groupVersion": "agones.dev/v1",
        "version": "v1"
    }
    ],
    "preferredVersion": {
    "groupVersion": "agones.dev/v1",
    "version": "v1"
    },
    "serverAddressByClientCIDRs": null
}

List Agones resources

curl http://localhost:8001/apis/agones.dev/v1

{
  "kind": "APIResourceList",
  "apiVersion": "v1",
  "groupVersion": "agones.dev/v1",
  "resources": [
    {
      "name": "gameservers",
      "singularName": "gameserver",
      "namespaced": true,
      "kind": "GameServer",
      "verbs": [
        "delete",
        "deletecollection",
        "get",
        "list",
        "patch",
        "create",
        "update",
        "watch"
      ],
      "shortNames": [
        "gs"
      ]
    }
  ]
}

List all gameservers in the default namespace

curl http://localhost:8001/apis/agones.dev/v1/namespaces/default/gameservers

{
    "apiVersion": "agones.dev/v1",
    "items": [
        {
            "apiVersion": "agones.dev/v1",
            "kind": "GameServer",
            "metadata": {
                "annotations": {
                    "kubectl.kubernetes.io/last-applied-configuration": "{\"apiVersion\":\"agones.dev/v1\",\"kind\":\"GameServer\",\"metadata\":{\"annotations\":{},\"name\":\"simple-game-server\",\"namespace\":\"default\"},\"spec\":{\"containerPort\":7654,\"hostPort\":7777,\"portPolicy\":\"static\",\"template\":{\"spec\":{\"containers\":[{\"image\":\"us-docker.pkg.dev/agones-images/examples/simple-game-server:0.31\",\"name\":\"simple-game-server\"}]}}}}\n"
                },
                "clusterName": "",
                "creationTimestamp": "2018-03-02T21:41:05Z",
                "finalizers": [
                    "agones.dev"
                ],
                "generation": 0,
                "name": "simple-game-server",
                "namespace": "default",
                "resourceVersion": "760",
                "selfLink": "/apis/agones.dev/v1/namespaces/default/gameservers/simple-game-server",
                "uid": "692beea6-1e62-11e8-beb2-080027637781"
            },
            "spec": {
                "PortPolicy": "Static",
                "container": "simple-game-server",
                "containerPort": 7654,
                "health": {
                    "failureThreshold": 3,
                    "initialDelaySeconds": 5,
                    "periodSeconds": 5
                },
                "hostPort": 7777,
                "protocol": "UDP",
                "template": {
                    "metadata": {
                        "creationTimestamp": null
                    },
                    "spec": {
                        "containers": [
                            {
                                "image": "us-docker.pkg.dev/agones-images/examples/simple-game-server:0.31",
                                "name": "simple-game-server",
                                "resources": {}
                            }
                        ]
                    }
                }
            },
            "status": {
                "address": "192.168.99.100",
                "nodeName": "agones",
                "port": 7777,
                "state": "Ready"
            }
        }
    ],
    "kind": "GameServerList",
    "metadata": {
        "continue": "",
        "resourceVersion": "1062",
        "selfLink": "/apis/agones.dev/v1/namespaces/default/gameservers"
    }
}

Allocate a gameserver from a fleet named ‘simple-game-server’, with GameServerAllocation

curl -d '{"apiVersion":"allocation.agones.dev/v1","kind":"GameServerAllocation","spec":{"required":{"matchLabels":{"agones.dev/fleet":"simple-game-server"}}}}' -H "Content-Type: application/json" -X POST http://localhost:8001/apis/allocation.agones.dev/v1/namespaces/default/gameserverallocations

{
    "kind": "GameServerAllocation",
    "apiVersion": "allocation.agones.dev/v1",
    "metadata": {
        "name": "simple-game-server-v6jwb-cmdcv",
        "namespace": "default",
        "creationTimestamp": "2019-07-03T17:19:47Z"
    },
    "spec": {
        "multiClusterSetting": {
            "policySelector": {}
        },
        "required": {
            "matchLabels": {
                "agones.dev/fleet": "simple-game-server"
            }
        },
        "scheduling": "Packed",
        "metadata": {}
    },
    "status": {
        "state": "Allocated",
        "gameServerName": "simple-game-server-v6jwb-cmdcv",
        "ports": [
            {
                "name": "default",
                "port": 7445
            }
        ],
        "address": "34.94.118.237",
        "nodeName": "gke-test-cluster-default-f11755a7-5km3"
    }
}

You may wish to review the Agones Kubernetes API for the full data structure reference.

The Kubernetes API Concepts section may also provide the more details on the API conventions that are used in the Kubernetes API.

Next Steps

Learn how to use Allocator Service for single and multi-cluster Allocation.

Docs: Troubleshooting

Thu, 03 Jan 2019 01:20:49 +0000

Something went wrong with my GameServer

If there is something going wrong with your GameServer, there are a few approaches to determining the cause:

Run with the local SDK server

A good first step for seeing what may be going wrong is replicating the issue locally. To do this you can take advantage of the Agones local SDK server , with the following troubleshooting steps:

Run your game server as a local binary against the local SDK server
Run your game server container against the local SDK server. It’s worth noting that running with docker run --network=host ... can be an easy way to allow your game server container(s) access to the local SDK server)

At each stage, keep an eye on the logs of your game server binary, and the local SDK server, and ensure there are no system errors.

Run as a `GameServer` rather than a `Fleet`

A Fleet will automatically replace any unhealthy GameServer under its control - which can make it hard to catch all the details to determine the cause.

To work around this, instantiate a single instance of your game server as a
GameServer within your Agones cluster.

This GameServer will not be replaced if it moves to an Unhealthy state, giving you time to introspect what is going wrong.

Introspect with Kubernetes tooling

There are many Kubernetes tools that will help with determining where things have potentially gone wrong for your game server. Here are a few you may want to try.

kubectl describe

Depending on what is happening, you may want to run kubectl describe <gameserver name> to view the events that are associated with that particular GameServer resource. This can give you insight into the lifecycle of the GameServer and if anything has gone wrong.

For example, here we can see where the simple-game-server example has been moved to the Unhealthy state due to a crash in the backing GameServer Pod container’s binary.

kubectl describe gs simple-game-server-zqppv

Name:         simple-game-server-zqppv
Namespace:    default
Labels:       <none>
Annotations:  agones.dev/sdk-version: 1.0.0-dce1546
API Version:  agones.dev/v1
Kind:         GameServer
Metadata:
  Creation Timestamp:  2019-08-16T21:25:44Z
  Finalizers:
    agones.dev
  Generate Name:     simple-game-server-
  Generation:        1
  Resource Version:  1378575
  Self Link:         /apis/agones.dev/v1/namespaces/default/gameservers/simple-game-server-zqppv
  UID:               6818adc7-c06c-11e9-8dbd-42010a8a0109
Spec:
  Container:  simple-game-server
  Health:
    Failure Threshold:      3
    Initial Delay Seconds:  5
    Period Seconds:         5
  Ports:
    Container Port:  7654
    Host Port:       7058
    Name:            default
    Port Policy:     Dynamic
    Protocol:        UDP
  Scheduling:        Packed
  Template:
    Metadata:
      Creation Timestamp:  <nil>
    Spec:
      Containers:
        Image:  us-docker.pkg.dev/agones-images/examples/simple-game-server:0.31
        Name:   simple-game-server
        Resources:
          Limits:
            Cpu:     20m
            Memory:  32Mi
          Requests:
            Cpu:     20m
            Memory:  32Mi
Status:
  Address:    35.230.59.117
  Node Name:  gke-test-cluster-default-590db5e4-4s6r
  Ports:
    Name:          default
    Port:          7058
  Reserved Until:  <nil>
  State:           Unhealthy
Events:
  Type     Reason          Age   From                   Message
  ----     ------          ----  ----                   -------
  Normal   PortAllocation  72s   gameserver-controller  Port allocated
  Normal   Creating        72s   gameserver-controller  Pod simple-game-server-zqppv created
  Normal   Scheduled       72s   gameserver-controller  Address and port populated
  Normal   RequestReady    67s   gameserver-sidecar     SDK state change
  Normal   Ready           66s   gameserver-controller  SDK.Ready() complete
  Warning  Unhealthy       34s   health-controller      Issue with Gameserver pod

The backing Pod has the same name as the GameServer - so it’s also worth looking at the details and events for the Pod to see if there are any issues there, such as restarts due to binary crashes etc.

For example, you can see the restart count on the us-docker.pkg.dev/agones-images/examples/simple-game-server:0.31 container is set to 1, due to the game server binary crash

kubectl describe pod simple-game-server-zqppv

Name:               simple-game-server-zqppv
Namespace:          default
Priority:           0
PriorityClassName:  <none>
Node:               gke-test-cluster-default-590db5e4-4s6r/10.138.0.23
Start Time:         Fri, 16 Aug 2019 21:25:44 +0000
Labels:             agones.dev/gameserver=simple-game-server-zqppv
                    agones.dev/role=gameserver
Annotations:        agones.dev/container: simple-game-server
                    agones.dev/sdk-version: 1.0.0-dce1546
                    cluster-autoscaler.kubernetes.io/safe-to-evict: false
Status:             Running
IP:                 10.48.1.80
Controlled By:      GameServer/simple-game-server-zqppv
Containers:
  simple-game-server:
    Container ID:   docker://69eacd03cc89b0636b78abe47926b02183ba84d18fa20649ca443f5232511661
    Image:          us-docker.pkg.dev/agones-images/examples/simple-game-server:0.31
    Image ID:       docker-pullable://gcr.io/agones-images/simple-game-server@sha256:6a60eff5e68b88b5ce75ae98082d79cff36cda411a090f3495760e5c3b6c3575
    Port:           7654/UDP
    Host Port:      7058/UDP
    State:          Running
      Started:      Fri, 16 Aug 2019 21:26:22 +0000
    Last State:     Terminated
      Reason:       Completed
      Exit Code:    0
      Started:      Fri, 16 Aug 2019 21:25:45 +0000
      Finished:     Fri, 16 Aug 2019 21:26:22 +0000
    Ready:          True
    Restart Count:  1
    Limits:
      cpu:     20m
      memory:  32Mi
    Requests:
      cpu:        20m
      memory:     32Mi
    Liveness:     http-get http://:8080/gshealthz delay=5s timeout=1s period=5s #success=1 #failure=3
    Environment:  <none>
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from empty (ro)
  agones-gameserver-sidecar:
    Container ID:   docker://f3c475c34d26232e19b60be65b03bc6ce41931f4c37e00770d3ab4a36281d31c
    Image:          gcr.io/agones-mark/agones-sdk:1.0.0-dce1546
    Image ID:       docker-pullable://gcr.io/agones-mark/agones-sdk@sha256:4b5693e95ee3023a2b2e2099d102bb6bac58d4ce0ac472e58a09cee6d160cd19
    Port:           <none>
    Host Port:      <none>
    State:          Running
      Started:      Fri, 16 Aug 2019 21:25:48 +0000
    Ready:          True
    Restart Count:  0
    Requests:
      cpu:     30m
    Liveness:  http-get http://:8080/healthz delay=3s timeout=1s period=3s #success=1 #failure=3
    Environment:
      GAMESERVER_NAME:  simple-game-server-zqppv
      POD_NAMESPACE:    default (v1:metadata.namespace)
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from agones-sdk-token-vr6qq (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             True
  ContainersReady   True
  PodScheduled      True
Volumes:
  empty:
    Type:    EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:
  agones-sdk-token-vr6qq:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  agones-sdk-token-vr6qq
    Optional:    false
QoS Class:       Burstable
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type    Reason     Age                   From                                             Message
  ----    ------     ----                  ----                                             -------
  Normal  Scheduled  2m32s                 default-scheduler                                Successfully assigned default/simple-game-server-zqppv to gke-test-cluster-default-590db5e4-4s6r
  Normal  Pulling    2m31s                 kubelet, gke-test-cluster-default-590db5e4-4s6r  pulling image "gcr.io/agones-mark/agones-sdk:1.0.0-dce1546"
  Normal  Started    2m28s                 kubelet, gke-test-cluster-default-590db5e4-4s6r  Started container
  Normal  Pulled     2m28s                 kubelet, gke-test-cluster-default-590db5e4-4s6r  Successfully pulled image "gcr.io/agones-mark/agones-sdk:1.0.0-dce1546"
  Normal  Created    2m28s                 kubelet, gke-test-cluster-default-590db5e4-4s6r  Created container
  Normal  Created    114s (x2 over 2m31s)  kubelet, gke-test-cluster-default-590db5e4-4s6r  Created container
  Normal  Started    114s (x2 over 2m31s)  kubelet, gke-test-cluster-default-590db5e4-4s6r  Started container
  Normal  Pulled     114s (x2 over 2m31s)  kubelet, gke-test-cluster-default-590db5e4-4s6r  Container image "us-docker.pkg.dev/agones-images/examples/simple-game-server:0.31" already present on machine

Finally, you can also get the logs of your GameServer Pod as well via kubectl logs <pod name> -c <game server container name>, for example:

kubectl logs simple-game-server-zqppv -c simple-game-server

2019/08/16 21:26:23 Creating SDK instance
2019/08/16 21:26:24 Starting Health Ping
2019/08/16 21:26:24 Starting UDP server, listening on port 7654
2019/08/16 21:26:24 Marking this server as ready

The above commands will only give the most recent container’s logs (so we won’t get the previous crash), but you can use kubectl logs --previous=true simple-game-server-zqppv -c simple-game-server to get the previous instance of the containers logs, or use your Kubernetes platform of choice’s logging aggregation tools to view the crash details.

kubectl events

The “Events” section that is seen at the bottom of a kubectl describe is backed an actual Event record in Kubernetes, which can be queried - and is general persistent for an hour after it is created.

Therefore, even a GameServer or Pod resource is no longer available in the system, its Events may well be.

kubectl get events can be used to see all these events. This can also be grepped with the GameServer name to see all events across both the GameServer and its backing Pod, like so:

kubectl get events | grep simple-game-server-v992s-jwpx2

2m47s       Normal   PortAllocation          gameserver/simple-game-server-v992s-jwpx2   Port allocated
2m47s       Normal   Creating                gameserver/simple-game-server-v992s-jwpx2   Pod simple-game-server-v992s-jwpx2 created
2m47s       Normal   Scheduled               pod/simple-game-server-v992s-jwpx2          Successfully assigned default/simple-game-server-v992s-jwpx2 to gke-test-cluster-default-77e7f57d-j1mp
2m47s       Normal   Scheduled               gameserver/simple-game-server-v992s-jwpx2   Address and port populated
2m46s       Normal   Pulled                  pod/simple-game-server-v992s-jwpx2          Container image "us-docker.pkg.dev/agones-images/examples/simple-game-server:0.31" already present on machine
2m46s       Normal   Created                 pod/simple-game-server-v992s-jwpx2          Created container simple-game-server
2m45s       Normal   Started                 pod/simple-game-server-v992s-jwpx2          Started container simple-game-server
2m45s       Normal   Pulled                  pod/simple-game-server-v992s-jwpx2          Container image "gcr.io/agones-images/agones-sdk:1.7.0" already present on machine
2m45s       Normal   Created                 pod/simple-game-server-v992s-jwpx2          Created container agones-gameserver-sidecar
2m45s       Normal   Started                 pod/simple-game-server-v992s-jwpx2          Started container agones-gameserver-sidecar
2m45s       Normal   RequestReady            gameserver/simple-game-server-v992s-jwpx2   SDK state change
2m45s       Normal   Ready                   gameserver/simple-game-server-v992s-jwpx2   SDK.Ready() complete
2m47s       Normal   SuccessfulCreate        gameserverset/simple-game-server-v992s      Created gameserver: simple-game-server-v992s-jwpx2

Other techniques

For more tips and tricks, the Kubernetes Cheatsheet: Interactive with Pods also provides more troubleshooting techniques.

How do I see the logs for Agones?

If something is going wrong, and you want to see the logs for Agones, there are potentially two places you will want to check:

The controller: assuming you installed Agones in the agones-system namespace, you will find that there is a single pod called agones-controller-<hash> (where hash is the unique code that Kubernetes generates) that exists there, that you can get the logs from. This is the main controller for Agones, and should be the first place to check when things go wrong.
1. To get the logs from this controller run:
  kubectl logs --namespace=agones-system agones-controller-<hash>
The SDK server sidecar: Agones runs a small gRPC + http server for the SDK in a container in the same network namespace as the game server container to connect to via the SDK.
The logs from this SDK server are also useful for tracking down issues, especially if you are having trouble with a particular GameServer.
1. To find the Pod for the GameServer look for the pod with a name that is prefixed with the name of the owning GameServer. For example if you have a GameServer named simple-game-server, it’s pod could potentially be named simple-game-server-dnbwj.
2. To get the logs from that Pod, we need to specify that we want the logs from the agones-gameserver-sidecar container. To do that, run the following:
  kubectl logs simple-game-server-dnbwj -c agones-gameserver-sidecar

Agones uses JSON structured logging, therefore errors will be visible through the "severity":"info" key and value.

Enable Debug Level Logging for the SDK Server

By default, the SDK Server binary is set to an Info level of logging.

You can use the sdkServer.logLevel to increase this to Debug levels, and see extra information about what is happening with the SDK Server that runs alonside your game server container(s).

See the GameServer reference for configuration details.

Enable Debug Level Logging for the Agones Controller

By default, the log level for the Agones controller is “info”. To get a more verbose log output, switch this to “debug” via the agones.controller.logLevel Helm Configuration parameters at installation.

The Feature Flag I enabled/disabled isn’t working as expected

It’s entirely possible that Alpha features may still have bugs in them (They are alpha after all 😃), but the first thing to check is what the actual Feature Flags states were passed to Agones are, and that they were set correctly.

The easiest way is to check the top info level log lines from the Agones controller.

For example:

$ kubectl logs -n agones-system agones-controller-7575dc59-7p2rg  | head
{"filename":"/home/agones/logs/agones-controller-20220615_211540.log","message":"logging to file","numbackups":99,"severity":"info","source":"main","time":"2022-06-15T21:15:40.309349789Z"}
{"logLevel":"info","message":"Setting LogLevel configuration","severity":"info","source":"main","time":"2022-06-15T21:15:40.309403296Z"}
{"ctlConf":{"MinPort":7000,"MaxPort":8000,"SidecarImage":"gcr.io/agones-images/agones-sdk:1.23.0","SidecarCPURequest":"30m","SidecarCPULimit":"0","SidecarMemoryRequest":"0","SidecarMemoryLimit":"0","SdkServiceAccount":"agones-sdk","AlwaysPullSidecar":false,"PrometheusMetrics":true,"Stackdriver":false,"StackdriverLabels":"","KeyFile":"/home/agones/certs/server.key","CertFile":"/home/agones/certs/server.crt","KubeConfig":"","GCPProjectID":"","NumWorkers":100,"APIServerSustainedQPS":400,"APIServerBurstQPS":500,"LogDir":"/home/agones/logs","LogLevel":"info","LogSizeLimitMB":10000},"featureGates":"Example=true\u0026NodeExternalDNS=true\u0026PlayerAllocationFilter=false\u0026PlayerTracking=false","message":"starting gameServer operator...","severity":"info","source":"main","time":"2022-06-15T21:15:40.309528802Z","version":"1.23.0"}
...

The ctlConf section has the full configuration for Agones as it was passed to the controller. Within that log line there is a featureGates key, that has the full Feature Gate configuration as a URL Query String (\u0026 is JSON for &), so you can see if the Feature Gates are set as expected.

I uninstalled Agones before deleted all my `GameServers` and now they won’t delete

Agones GameServers use Finalizers to manage garbage collection of the GameServers. This means that if the Agones controller doesn’t remove the finalizer for you (i.e. if it has been uninstalled), it can be tricky to remove them all.

Thankfully, if we create a patch to remove the finalizers from all GameServers, we can delete them with impunity.

A quick one liner to do this:

kubectl get gameserver -o name | xargs -n1 -P1 -I{} kubectl patch {} --type=merge -p '{"metadata": {"finalizers": []}}'

Once this is done, you can kubectl delete gs --all and clean everything up (if it’s not gone already).

I’m getting Forbidden errors when trying to install Agones

Ensure that you are running Kubernetes 1.12 or later, which does not require any special clusterrolebindings to install Agones.

If you want to install Agones on an older version of Kubernetes, you need to create a clusterrolebinding to add your identity as a cluster admin, e.g.

# Kubernetes Engine
kubectl create clusterrolebinding cluster-admin-binding \
  --clusterrole cluster-admin --user `gcloud config get-value account`
# Minikube
kubectl create clusterrolebinding cluster-admin-binding \
  --clusterrole=cluster-admin --serviceaccount=kube-system:default

On GKE, gcloud config get-value accounts will return a lowercase email address, so if you are using a CamelCase email, you may need to type it in manually.

I’m getting stuck in “Terminating” when I uninstall Agones

If you try to uninstall the agones-system namespace before you have removed all of the components in the namespace you may end up in a Terminating state.

kubectl get ns

NAME              STATUS        AGE                                                                                                                                                    
agones-system     Terminating   4d

Fixing this up requires us to bypass the finalizer in Kubernetes (article link), by manually changing the namespace details:

First get the current state of the namespace:

kubectl get namespace agones-system -o json >tmp.json

Edit the response tmp.json to remove the finalizer data, for example remove the following:

"spec": {
    "finalizers": [
        "kubernetes"
    ]
},

Open a new terminal to proxy traffic:

 kubectl proxy

Starting to serve on 127.0.0.1:8001

Now make an API call to send the altered namespace data:

 curl -k -H "Content-Type: application/json" -X PUT --data-binary @tmp.json http://127.0.0.1:8001/api/v1/namespaces/agones-system/finalize

You may need to clean up any other Agones related resources you have in your cluster at this point.