Thread Rating:
  • 1 Vote(s) - 5 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Dev Diary 17 :: 2016 :: March
#1
Hey Folks, 

I'm going to give an assortment of "What's going on" here, both good and bad.

The Good
Considering 'the bad' right now, updates have been occurring at a good pace again. I think once we handle a few of 'the bad' we can stay very consistent in the frequent updates/changes to the games and really start pushing content plans ahead.

We have an offline gains trial coming up.

We have the new Skill Map system coming up soon with major pet rebalancing.

We have a new monthly event coming up.

The Bad
A lot of players are praising the speed of things and talking about how updates are coming along, but i'm going to side a bit with the negative nancies here for a moment. Things are NOT going even close to as stable as we had hoped by this point in time. 

The migration went very bumpy during the development process and it hasn't been going all that great afterwards either on our end. We have a lot of errors/problems that are not the "Just google it" kind. We have pretty much no way to truly debug issues that are not easily reproducable due to the server set up on a backend server that unfortunately does not match our rather unique needs for this game.

Many players are unable to connect due to the new ports used.

Groups of players are experiencing lag, or bizarre issues (such as minor rollbacks) caused by OS-level issues in certain dedicated servers.

The resource usage on the current servers are not what they should be based on our self-hosting tests.

The biggest bad: The next community discussion (and thus the rest of the discussion-results list) is going to be delayed a weak, aiming now at March 19th

The Solution (Nerd time)
We had a rather incredible, and important meeting to discuss the fact that I simply do not want to be tackling tickets and technical issues that are not exactly normal for developers. This has resulted in the simple fact that we have to make some serious changes to improve development and reduce the stress on the community and ourselves.

Time has been spent completely redesigning the server and client while still trying to tackle these recent updates and planning for the future. I'm happy to say a good amount of progress has already been made, and we are actually already well on our way, going through preliminary testing phases on the new design.

The new design entails listener servers that pass on connections from port 80 (hello work & campus internet gaming again), and creates an instance-level of the servers instead of running each instance as its own server application. This will significantly save on CPU & Memory usage to the point that we can ensure far more power than we'll need for the servers and at a fraction of the cost.

In our new design:
-No reliance on backend hosting solutions (other than your typical AWS)
-Much faster production/development with very easy test phasing compared to the current design
-Much faster maintenances (potentially as fast as they used to be...60 seconds)
-Ability to perform global broadcasts (finally - much needed)
-Ability to debug and monitor specific rooms in real-time
-Ability to shut down instances that may be broken & hindering the service (causing the lag some groups are getting)


None of this screams excitement for content-hungry folks, but we should be at a public testable version by this weekend where we can keep moving forward again. Some of our original designs simply didn't work out and it was difficult to prepare perfectly for such a massive migration without seeing the information/problems first hand. We are now rectifying that and the purpose of this post is to let you know that the recent lack of publication has been due to spending a lot of efforts in ensuring that we had a solid (and possible) plan before communicating what we were going to do.

Ray
Reply
#2
Hi folks,

Now that we are finally getting to a more stable state with the new server design (Don't jinx it!), I'd like to finally put together a bunch of rant/love/hate/explanation as to what has been going on over the last few months for those interested in our reasoning for leaving Yahoo, why things went downhill, and more.

The Game and the Unexpected
I worked on IOU back around September/October of 2014 as I grew frustrated in how Idle games were constantly losing save data...except I love co-op and multiplayer games, so I went a few (hundred) steps further. I was in school, and living in the most unemployed city in my country, had very little I could invest with, and my goals were only to put together a fun game and see how people took it.

I released it without any real publication back in January 15th 2015, and with word of mouth the game grew very quickly. It was doing WAY better than I imagined, growing hundreds of users...all seemingly by word of mouth.

I contacted Kongregate about it and they didn't think the game would do well, and were reluctant about it. I conveyed some facts and still nothing moved forward until finally being contacted again about putting it into beta on Kongregate. Which we did late May/June I believe.

From there the game went haywire...after going public we had more players than I could ever keep up with or ever dreamed of.

Yahoo
PlayerIO is an incredible service that simply does things better than any other host did for multiplayer web-games. I never found anything that could compete with the simplicity and power of this service....HOWEVER, Yahoo bought them out and it went downhill very quickly ever since. 

The issue was that Yahoo handled ALL the scaling and backend service so well that I had a passive realiance with them, but as things grew, yahoo failed, and between failing to filter ddos attacks, and server outages, and poor support (and promises to fix issues that months later were never fixed) it became a priority to move off of them. I knew well before they announced the shutdown of Yahoo Games that that is where they were headed quickly.

Playfab was suggested, and after having facetime with the CEO, and being assured their service was a good bet for migration, I jumped into development head first in October 2015. 

The issue was that Yahoo handled EVERYTHING, and Playfab, like all the other servers, did not handle things the same way, which meant completely having to re-program the client and server for IOU entirely to fit the new backend service.

A lot of people don't understand that this is a 2 man team trying to maintain servers that are intended to host thousands of rooms and more thousands of players. This kind of traffic wasn't expected and the resources to get a bigger team together where we live is just not feasible. 

Playfab
After months of preparing the full migration and rewriting the entire game, so many issues were learned in terms of that Playfab was not the best bet for IOU. Playfab is a great service for many different kinds of games, but IOU has very unique server needs, and after working around all the problems we encountered in the rewrite, we released the server publicly with Playfab as the backend.

Of course testing with a good group of players compared to going public with thousands of users is entirely different and every single problem arised. 

Dedicated servers costing a fortune couldn't handle 80+ rooms/players, our new database was bogging down because of .nets connection pooling overloading things, and we found many more backend issues out of our control (just like Yahoo - except on a responsive service with a great CEO).

The problem was that this backend service obviously can't change how everything works just for our sake, so I got more desperate than i've ever been in my life after working all day every single day since I released this game, and i'd like to imagine a day where i'm not stressed to the max with upset and disappointed players on a game I once loved working on - and the fact that I would rather enjoy being able to take my kids camping or -something- this year.

Meeting
I called Jay for a meeting to discuss the fact that either we find a solution or IOU is going to die because I couldn't handle working 14-16+ hours every day and not seemingly get anywhere with the same problems we had with Yahoo. I came up with a quick design for designing our own self-hosted backend service, and we were to work on that full-time from there on - which meant no updates/patches yet again, but it was either that for give the bad news to the community.

New Design
We had many players not understanding the reasoning or stresses on our end, iterating ignorant comments as if we had purposes done everything in our power to break the game and acting as if we were doing nothing to resolve things. Irregardless we managed to put together a full redesign for the second time, with backend service features (that usually takes full teams/companies months or years) within 3 weeks. 

Again, testing goes well, public release goes horrible, and while that was stressful and certain groups of people maintained "this game is going to die", for once I ignored and felt 100% confident because now when we ran into issues, we could actually fix them. There is NOTHING out of our control now.

Today finally the servers ran smooth and FAST, after many 18-20 hour straight days to the point of health risks, with persistence in not worrying about what negative nancies had to say and worrying about the long-term. This game will survive now for MANY years with this design working - cost effective, powerful, scalable, and no reliance on a backend service where we have no control.

I am over 6 months behind on goals, and have a plateful that could reach the ceiling, but we're moving forward now

My Comments


  1. I really am sorry for those who have been so invested in the game and had greater expectations. I have an unhealthy attachment to this game and while it's a 1st world problem, I have stressed more with this game and trying to deal with ridiculous issues that I never anticipated than anything in my life. It's just a game, but I love the hell out of working on it and the community within it to a fault.
  2. After being so heavily burned by the last several months, I have a new attitude. I'm not burnt out with the game, but i'm burnt out with making myself available on the front-lines to negative nancies when I don't need to, just to be insulted, branded or having to deal with ignorant rumors by a very small group of individuals. 

    I will no longer offer my time trying to make consistently irate people happy, nor put up with ignorant comments towards my team. I don't care if people have spent money or many months on the game. The support offered is for the community...if you have invested into packs or have played 300 days, it makes no difference...we will offer the best support we can - but trying to use that as leverage to push anger tactics will now just be ignored as it's childish and nothing positive can come of it.

    Support and long-term attendance has ensured this game's future, but it's future definitely doesn't rely on poisonous players who only come around to create tickets of abuse, or broadcast nothing but negativity in our chat.

    I never want to see great members of IOU's community leave, but if you feel you'll never be happy with the game, what can I do for you?
  3. I will try to do what's right for the game and attempt to significantly alleviate the lack of updates to the best of my ability. Can I make up for how incredibly patient most of you have been? I can't promise that, but I will continue being fully dedicated to this game and community.
  4. After server design maintenances, I notice a burst of upset players within minutes of not being able to log into the game, among other issues. Please understand these are massive changes and we EXPECT problems during these trials - you should too (Look at diablo 3's release, and that's from a billion dollar, massive manned company - it's not simple unfortunately and we are game developers...jack of many things but impossible to be perfect at the many different types of skills required to make a game and concurrent server programming is not a walk in the park). I highly suggest waiting a while before logging in after the game is initially brought up to avoid having to jump through hoops to log in if you're becoming frustrated. We are finally getting somewhere with relogs, offline gains, etc, so please bear with us or simply expect up and down service until then.

What the plans are (In no specific order)
  1. Offline Gains - terrible timing if things are growing more stable, but it's still the plan, and a portion of this system is already complete
  2. DB Migration - We use amazon aurora, and it's not going great for scalability, or stability. Once we swap this, stability and performance should increase significantly
  3. Pet rebalancing & new PP shop with a unique new upgrade map
  4. Event
  5. Guild upkeep is backburnered until I can present something more favorable
  6. Fishing
  7. Ship system (I think this will be the first more major content addition - new game-mode and more)
  8. 1st permanent in-game Community Event - Sunday Bingo
  9. Have an art-house working on new assets, and a new UI design...going to take a while but it's delegated work anyways
  10. New challenge (2-parter!)
  11. Scheduled Community Discussions again
If you're expecting a major new content system in the next week; that's unrealistic, but it's definitely in the workflow at this point.

Thanks for the read, and thanks for having stuck around IOU all this time Smile

Ray
Reply


Forum Jump:


Users browsing this thread: 1 Guest(s)