Gameforge is a leading game developer of free-to-play massive online multiplayer (MMO) style games. It provides over 20 games to 450 million registered users on more than 20,000 servers worldwide. As Gameforge continues to grow, the challenges with server maintenance and communication between developers and system administrators continues to increase. I recently spoke with Felix Oechsler, Lead Windows System Administrator to look at the ways Gameforge is overcoming those challenges utilizing DevOps tools and methodology.
I thought it would be interesting to capture the implementation of DevOps at a gaming company since I haven’t seen much on gaming here at DevOps.com, so I took it upon myself to speak with Felix.
As a sys admin, there are two main systems at Gameforge, the windows and linux systems. Felix’s focus is on managing the servers and his job summary includes server upkeep, security, performance management, log data, and to oversee the health of the servers. When I asked Felix how Gameforge came to use DevOps, he told me that it started 2 years ago on a project they were working on. Many issues arose because the communication between the system administrators and developers was not efficient in regards to dumps and data logs. When an error had occurred they needed to inform the developers and then manually hand the data from the server over to the developer team to fix the issues. This took away valuable time from Felix and his fellow sys admins, and also caused various security issues, since they could not give server access to the developers. Realizing the teams needed a way to interact more efficiently and effectively they asked the simple question, how can we do this better?
“We needed a better log management solution to quickly and easily troubleshoot issues before our gamers noticed any problems in the games or systems. We also wanted an all-in-one solution that would be easy to use and manage across our different departments.”
-Felix Oechsler, Lead Windows System Adminstrator, Gameforge
Alas! Devops to the rescue!
Felix’s team began to implement devops tools and best practices such as Graylog’s centralized log management platform. This allowed developers to see the logs in one centralized location without involving the sysadmins to comb through the logs to deliver the data to the developers, freeing up valuable time of sysadmins and minimizing security issues. Graylog helps to deliver 5 beneficial uses including user access management, intuitive search and filtering, visualizations, alerting, and flexible log data extraction. Explanations on each of these can be found here, a case study done by Graylog.
“My development team created our own streams and set alerts for the games we were working on. We’re able to get real time alerts on critical errors in games and applications so we can react faster than before. In one case, our mobile game team received an alert that our connection to the Apple payment system was down. We were able to resolve the problem quickly and avoided the bad user experience of a payment not going through.”
– Dennis Simon, Lead Developer, Gameforge
Gameforge also begun using key performance indicators (KPIs), Graphite and Grafana, to see how the applications are performing, such as high CPU costs. Using both Graylog and the KPIs, developers were able to continuous deploy updates to the games without involving the sysadmins, cutting down on time and cost, and effectively creating a better playing environment for their gamers. Gameforge has developed their own personal, customized continuous deployment platform using C# and powershare to code the mechanism. They decided to develop the platform in-house to best fit their structure and security policies.
Since Gameforge has begun implementing devops tools and best practices they have seen great beneficial change.
“We now monitor our entire server environment with Graylog and have total visibility, enabling us to search for internal application problems across multiple backend servers. Also, our Developers and DevOps people rely on Graylog’s alerting functionality to react to critical errors in our games or applications. Using Graylog, we are able to provide a better user experience for our gamers. Our reaction time is much faster than before, and our players experience less outages and other error-related problems.”
– Felix Oechsler, Lead Windows System Administrator, Gameforge
Using DevOps, Gameforge has freed up valuable time for the system administrators and provided immense visibility to the development teams. Visibility provides quick access to alerts on errors, allowing problems to be fixed quicker and pushed live via a continuous deployment environment. It is safe to say that devops is a valuable methodology for system administrators and developers alike.