Dev Diary 17 :: 2016 :: March - Printable Version +- Idle Online Universe RPG (https://iourpg.com/forum) +-- Forum: Development (https://iourpg.com/forum/forumdisplay.php?fid=1) +--- Forum: Dev Blog (https://iourpg.com/forum/forumdisplay.php?fid=3) +--- Thread: Dev Diary 17 :: 2016 :: March (/showthread.php?tid=2011) |
Dev Diary 17 :: 2016 :: March - Ray - 03-10-2016 Hey Folks, I'm going to give an assortment of "What's going on" here, both good and bad. The Good Considering 'the bad' right now, updates have been occurring at a good pace again. I think once we handle a few of 'the bad' we can stay very consistent in the frequent updates/changes to the games and really start pushing content plans ahead. We have an offline gains trial coming up. We have the new Skill Map system coming up soon with major pet rebalancing. We have a new monthly event coming up. The Bad A lot of players are praising the speed of things and talking about how updates are coming along, but i'm going to side a bit with the negative nancies here for a moment. Things are NOT going even close to as stable as we had hoped by this point in time. The migration went very bumpy during the development process and it hasn't been going all that great afterwards either on our end. We have a lot of errors/problems that are not the "Just google it" kind. We have pretty much no way to truly debug issues that are not easily reproducable due to the server set up on a backend server that unfortunately does not match our rather unique needs for this game. Many players are unable to connect due to the new ports used. Groups of players are experiencing lag, or bizarre issues (such as minor rollbacks) caused by OS-level issues in certain dedicated servers. The resource usage on the current servers are not what they should be based on our self-hosting tests. The biggest bad: The next community discussion (and thus the rest of the discussion-results list) is going to be delayed a weak, aiming now at March 19th The Solution (Nerd time) We had a rather incredible, and important meeting to discuss the fact that I simply do not want to be tackling tickets and technical issues that are not exactly normal for developers. This has resulted in the simple fact that we have to make some serious changes to improve development and reduce the stress on the community and ourselves. Time has been spent completely redesigning the server and client while still trying to tackle these recent updates and planning for the future. I'm happy to say a good amount of progress has already been made, and we are actually already well on our way, going through preliminary testing phases on the new design. The new design entails listener servers that pass on connections from port 80 (hello work & campus internet gaming again), and creates an instance-level of the servers instead of running each instance as its own server application. This will significantly save on CPU & Memory usage to the point that we can ensure far more power than we'll need for the servers and at a fraction of the cost. In our new design: -No reliance on backend hosting solutions (other than your typical AWS) -Much faster production/development with very easy test phasing compared to the current design -Much faster maintenances (potentially as fast as they used to be...60 seconds) -Ability to perform global broadcasts (finally - much needed) -Ability to debug and monitor specific rooms in real-time -Ability to shut down instances that may be broken & hindering the service (causing the lag some groups are getting) None of this screams excitement for content-hungry folks, but we should be at a public testable version by this weekend where we can keep moving forward again. Some of our original designs simply didn't work out and it was difficult to prepare perfectly for such a massive migration without seeing the information/problems first hand. We are now rectifying that and the purpose of this post is to let you know that the recent lack of publication has been due to spending a lot of efforts in ensuring that we had a solid (and possible) plan before communicating what we were going to do. Ray A guide to understanding the last few months - Ray - 03-24-2016 Hi folks, Now that we are finally getting to a more stable state with the new server design (Don't jinx it!), I'd like to finally put together a bunch of rant/love/hate/explanation as to what has been going on over the last few months for those interested in our reasoning for leaving Yahoo, why things went downhill, and more. The Game and the Unexpected I worked on IOU back around September/October of 2014 as I grew frustrated in how Idle games were constantly losing save data...except I love co-op and multiplayer games, so I went a few (hundred) steps further. I was in school, and living in the most unemployed city in my country, had very little I could invest with, and my goals were only to put together a fun game and see how people took it. I released it without any real publication back in January 15th 2015, and with word of mouth the game grew very quickly. It was doing WAY better than I imagined, growing hundreds of users...all seemingly by word of mouth. I contacted Kongregate about it and they didn't think the game would do well, and were reluctant about it. I conveyed some facts and still nothing moved forward until finally being contacted again about putting it into beta on Kongregate. Which we did late May/June I believe. From there the game went haywire...after going public we had more players than I could ever keep up with or ever dreamed of. Yahoo PlayerIO is an incredible service that simply does things better than any other host did for multiplayer web-games. I never found anything that could compete with the simplicity and power of this service....HOWEVER, Yahoo bought them out and it went downhill very quickly ever since. The issue was that Yahoo handled ALL the scaling and backend service so well that I had a passive realiance with them, but as things grew, yahoo failed, and between failing to filter ddos attacks, and server outages, and poor support (and promises to fix issues that months later were never fixed) it became a priority to move off of them. I knew well before they announced the shutdown of Yahoo Games that that is where they were headed quickly. Playfab was suggested, and after having facetime with the CEO, and being assured their service was a good bet for migration, I jumped into development head first in October 2015. The issue was that Yahoo handled EVERYTHING, and Playfab, like all the other servers, did not handle things the same way, which meant completely having to re-program the client and server for IOU entirely to fit the new backend service. A lot of people don't understand that this is a 2 man team trying to maintain servers that are intended to host thousands of rooms and more thousands of players. This kind of traffic wasn't expected and the resources to get a bigger team together where we live is just not feasible. Playfab After months of preparing the full migration and rewriting the entire game, so many issues were learned in terms of that Playfab was not the best bet for IOU. Playfab is a great service for many different kinds of games, but IOU has very unique server needs, and after working around all the problems we encountered in the rewrite, we released the server publicly with Playfab as the backend. Of course testing with a good group of players compared to going public with thousands of users is entirely different and every single problem arised. Dedicated servers costing a fortune couldn't handle 80+ rooms/players, our new database was bogging down because of .nets connection pooling overloading things, and we found many more backend issues out of our control (just like Yahoo - except on a responsive service with a great CEO). The problem was that this backend service obviously can't change how everything works just for our sake, so I got more desperate than i've ever been in my life after working all day every single day since I released this game, and i'd like to imagine a day where i'm not stressed to the max with upset and disappointed players on a game I once loved working on - and the fact that I would rather enjoy being able to take my kids camping or -something- this year. Meeting I called Jay for a meeting to discuss the fact that either we find a solution or IOU is going to die because I couldn't handle working 14-16+ hours every day and not seemingly get anywhere with the same problems we had with Yahoo. I came up with a quick design for designing our own self-hosted backend service, and we were to work on that full-time from there on - which meant no updates/patches yet again, but it was either that for give the bad news to the community. New Design We had many players not understanding the reasoning or stresses on our end, iterating ignorant comments as if we had purposes done everything in our power to break the game and acting as if we were doing nothing to resolve things. Irregardless we managed to put together a full redesign for the second time, with backend service features (that usually takes full teams/companies months or years) within 3 weeks. Again, testing goes well, public release goes horrible, and while that was stressful and certain groups of people maintained "this game is going to die", for once I ignored and felt 100% confident because now when we ran into issues, we could actually fix them. There is NOTHING out of our control now. Today finally the servers ran smooth and FAST, after many 18-20 hour straight days to the point of health risks, with persistence in not worrying about what negative nancies had to say and worrying about the long-term. This game will survive now for MANY years with this design working - cost effective, powerful, scalable, and no reliance on a backend service where we have no control. I am over 6 months behind on goals, and have a plateful that could reach the ceiling, but we're moving forward now. My Comments
What the plans are (In no specific order)
Thanks for the read, and thanks for having stuck around IOU all this time Ray |