03-24-2016, 07:03 PM
Hi folks,
Now that we are finally getting to a more stable state with the new server design (Don't jinx it!), I'd like to finally put together a bunch of rant/love/hate/explanation as to what has been going on over the last few months for those interested in our reasoning for leaving Yahoo, why things went downhill, and more.
The Game and the Unexpected
I worked on IOU back around September/October of 2014 as I grew frustrated in how Idle games were constantly losing save data...except I love co-op and multiplayer games, so I went a few (hundred) steps further. I was in school, and living in the most unemployed city in my country, had very little I could invest with, and my goals were only to put together a fun game and see how people took it.
I released it without any real publication back in January 15th 2015, and with word of mouth the game grew very quickly. It was doing WAY better than I imagined, growing hundreds of users...all seemingly by word of mouth.
I contacted Kongregate about it and they didn't think the game would do well, and were reluctant about it. I conveyed some facts and still nothing moved forward until finally being contacted again about putting it into beta on Kongregate. Which we did late May/June I believe.
From there the game went haywire...after going public we had more players than I could ever keep up with or ever dreamed of.
Yahoo
PlayerIO is an incredible service that simply does things better than any other host did for multiplayer web-games. I never found anything that could compete with the simplicity and power of this service....HOWEVER, Yahoo bought them out and it went downhill very quickly ever since.
The issue was that Yahoo handled ALL the scaling and backend service so well that I had a passive realiance with them, but as things grew, yahoo failed, and between failing to filter ddos attacks, and server outages, and poor support (and promises to fix issues that months later were never fixed) it became a priority to move off of them. I knew well before they announced the shutdown of Yahoo Games that that is where they were headed quickly.
Playfab was suggested, and after having facetime with the CEO, and being assured their service was a good bet for migration, I jumped into development head first in October 2015.
The issue was that Yahoo handled EVERYTHING, and Playfab, like all the other servers, did not handle things the same way, which meant completely having to re-program the client and server for IOU entirely to fit the new backend service.
A lot of people don't understand that this is a 2 man team trying to maintain servers that are intended to host thousands of rooms and more thousands of players. This kind of traffic wasn't expected and the resources to get a bigger team together where we live is just not feasible.
Playfab
After months of preparing the full migration and rewriting the entire game, so many issues were learned in terms of that Playfab was not the best bet for IOU. Playfab is a great service for many different kinds of games, but IOU has very unique server needs, and after working around all the problems we encountered in the rewrite, we released the server publicly with Playfab as the backend.
Of course testing with a good group of players compared to going public with thousands of users is entirely different and every single problem arised.
Dedicated servers costing a fortune couldn't handle 80+ rooms/players, our new database was bogging down because of .nets connection pooling overloading things, and we found many more backend issues out of our control (just like Yahoo - except on a responsive service with a great CEO).
The problem was that this backend service obviously can't change how everything works just for our sake, so I got more desperate than i've ever been in my life after working all day every single day since I released this game, and i'd like to imagine a day where i'm not stressed to the max with upset and disappointed players on a game I once loved working on - and the fact that I would rather enjoy being able to take my kids camping or -something- this year.
Meeting
I called Jay for a meeting to discuss the fact that either we find a solution or IOU is going to die because I couldn't handle working 14-16+ hours every day and not seemingly get anywhere with the same problems we had with Yahoo. I came up with a quick design for designing our own self-hosted backend service, and we were to work on that full-time from there on - which meant no updates/patches yet again, but it was either that for give the bad news to the community.
New Design
We had many players not understanding the reasoning or stresses on our end, iterating ignorant comments as if we had purposes done everything in our power to break the game and acting as if we were doing nothing to resolve things. Irregardless we managed to put together a full redesign for the second time, with backend service features (that usually takes full teams/companies months or years) within 3 weeks.
Again, testing goes well, public release goes horrible, and while that was stressful and certain groups of people maintained "this game is going to die", for once I ignored and felt 100% confident because now when we ran into issues, we could actually fix them. There is NOTHING out of our control now.
Today finally the servers ran smooth and FAST, after many 18-20 hour straight days to the point of health risks, with persistence in not worrying about what negative nancies had to say and worrying about the long-term. This game will survive now for MANY years with this design working - cost effective, powerful, scalable, and no reliance on a backend service where we have no control.
I am over 6 months behind on goals, and have a plateful that could reach the ceiling, but we're moving forward now.
My Comments
What the plans are (In no specific order)
Thanks for the read, and thanks for having stuck around IOU all this time
Ray
Now that we are finally getting to a more stable state with the new server design (Don't jinx it!), I'd like to finally put together a bunch of rant/love/hate/explanation as to what has been going on over the last few months for those interested in our reasoning for leaving Yahoo, why things went downhill, and more.
The Game and the Unexpected
I worked on IOU back around September/October of 2014 as I grew frustrated in how Idle games were constantly losing save data...except I love co-op and multiplayer games, so I went a few (hundred) steps further. I was in school, and living in the most unemployed city in my country, had very little I could invest with, and my goals were only to put together a fun game and see how people took it.
I released it without any real publication back in January 15th 2015, and with word of mouth the game grew very quickly. It was doing WAY better than I imagined, growing hundreds of users...all seemingly by word of mouth.
I contacted Kongregate about it and they didn't think the game would do well, and were reluctant about it. I conveyed some facts and still nothing moved forward until finally being contacted again about putting it into beta on Kongregate. Which we did late May/June I believe.
From there the game went haywire...after going public we had more players than I could ever keep up with or ever dreamed of.
Yahoo
PlayerIO is an incredible service that simply does things better than any other host did for multiplayer web-games. I never found anything that could compete with the simplicity and power of this service....HOWEVER, Yahoo bought them out and it went downhill very quickly ever since.
The issue was that Yahoo handled ALL the scaling and backend service so well that I had a passive realiance with them, but as things grew, yahoo failed, and between failing to filter ddos attacks, and server outages, and poor support (and promises to fix issues that months later were never fixed) it became a priority to move off of them. I knew well before they announced the shutdown of Yahoo Games that that is where they were headed quickly.
Playfab was suggested, and after having facetime with the CEO, and being assured their service was a good bet for migration, I jumped into development head first in October 2015.
The issue was that Yahoo handled EVERYTHING, and Playfab, like all the other servers, did not handle things the same way, which meant completely having to re-program the client and server for IOU entirely to fit the new backend service.
A lot of people don't understand that this is a 2 man team trying to maintain servers that are intended to host thousands of rooms and more thousands of players. This kind of traffic wasn't expected and the resources to get a bigger team together where we live is just not feasible.
Playfab
After months of preparing the full migration and rewriting the entire game, so many issues were learned in terms of that Playfab was not the best bet for IOU. Playfab is a great service for many different kinds of games, but IOU has very unique server needs, and after working around all the problems we encountered in the rewrite, we released the server publicly with Playfab as the backend.
Of course testing with a good group of players compared to going public with thousands of users is entirely different and every single problem arised.
Dedicated servers costing a fortune couldn't handle 80+ rooms/players, our new database was bogging down because of .nets connection pooling overloading things, and we found many more backend issues out of our control (just like Yahoo - except on a responsive service with a great CEO).
The problem was that this backend service obviously can't change how everything works just for our sake, so I got more desperate than i've ever been in my life after working all day every single day since I released this game, and i'd like to imagine a day where i'm not stressed to the max with upset and disappointed players on a game I once loved working on - and the fact that I would rather enjoy being able to take my kids camping or -something- this year.
Meeting
I called Jay for a meeting to discuss the fact that either we find a solution or IOU is going to die because I couldn't handle working 14-16+ hours every day and not seemingly get anywhere with the same problems we had with Yahoo. I came up with a quick design for designing our own self-hosted backend service, and we were to work on that full-time from there on - which meant no updates/patches yet again, but it was either that for give the bad news to the community.
New Design
We had many players not understanding the reasoning or stresses on our end, iterating ignorant comments as if we had purposes done everything in our power to break the game and acting as if we were doing nothing to resolve things. Irregardless we managed to put together a full redesign for the second time, with backend service features (that usually takes full teams/companies months or years) within 3 weeks.
Again, testing goes well, public release goes horrible, and while that was stressful and certain groups of people maintained "this game is going to die", for once I ignored and felt 100% confident because now when we ran into issues, we could actually fix them. There is NOTHING out of our control now.
Today finally the servers ran smooth and FAST, after many 18-20 hour straight days to the point of health risks, with persistence in not worrying about what negative nancies had to say and worrying about the long-term. This game will survive now for MANY years with this design working - cost effective, powerful, scalable, and no reliance on a backend service where we have no control.
I am over 6 months behind on goals, and have a plateful that could reach the ceiling, but we're moving forward now.
My Comments
- I really am sorry for those who have been so invested in the game and had greater expectations. I have an unhealthy attachment to this game and while it's a 1st world problem, I have stressed more with this game and trying to deal with ridiculous issues that I never anticipated than anything in my life. It's just a game, but I love the hell out of working on it and the community within it to a fault.
- After being so heavily burned by the last several months, I have a new attitude. I'm not burnt out with the game, but i'm burnt out with making myself available on the front-lines to negative nancies when I don't need to, just to be insulted, branded or having to deal with ignorant rumors by a very small group of individuals.
I will no longer offer my time trying to make consistently irate people happy, nor put up with ignorant comments towards my team. I don't care if people have spent money or many months on the game. The support offered is for the community...if you have invested into packs or have played 300 days, it makes no difference...we will offer the best support we can - but trying to use that as leverage to push anger tactics will now just be ignored as it's childish and nothing positive can come of it.
Support and long-term attendance has ensured this game's future, but it's future definitely doesn't rely on poisonous players who only come around to create tickets of abuse, or broadcast nothing but negativity in our chat.
I never want to see great members of IOU's community leave, but if you feel you'll never be happy with the game, what can I do for you? - I will try to do what's right for the game and attempt to significantly alleviate the lack of updates to the best of my ability. Can I make up for how incredibly patient most of you have been? I can't promise that, but I will continue being fully dedicated to this game and community.
- After server design maintenances, I notice a burst of upset players within minutes of not being able to log into the game, among other issues. Please understand these are massive changes and we EXPECT problems during these trials - you should too (Look at diablo 3's release, and that's from a billion dollar, massive manned company - it's not simple unfortunately and we are game developers...jack of many things but impossible to be perfect at the many different types of skills required to make a game and concurrent server programming is not a walk in the park). I highly suggest waiting a while before logging in after the game is initially brought up to avoid having to jump through hoops to log in if you're becoming frustrated. We are finally getting somewhere with relogs, offline gains, etc, so please bear with us or simply expect up and down service until then.
What the plans are (In no specific order)
- Offline Gains - terrible timing if things are growing more stable, but it's still the plan, and a portion of this system is already complete
- DB Migration - We use amazon aurora, and it's not going great for scalability, or stability. Once we swap this, stability and performance should increase significantly
- Pet rebalancing & new PP shop with a unique new upgrade map
- Event
- Guild upkeep is backburnered until I can present something more favorable
- Fishing
- Ship system (I think this will be the first more major content addition - new game-mode and more)
- 1st permanent in-game Community Event - Sunday Bingo
- Have an art-house working on new assets, and a new UI design...going to take a while but it's delegated work anyways
- New challenge (2-parter!)
- Scheduled Community Discussions again
Thanks for the read, and thanks for having stuck around IOU all this time
Ray