[VC] News January 7 - Cache Server Issues

We had several cache servers die today and they've since been replaced. This was an unplanned outage. Unfortunately the following information has been reset:

• Game History Feed Reset
• Rivals List Reset
• Counter Attack Costing Reset
• Bounty Cost Increase Reset
• Guild Feed Reset
• Gift Counter for the day Reset
• Misc Help Counters for the day
• Alerts will be re-shown
• Boss min dmg sliders will Reset
• Elite Chieftain Requests will expire

We are working on new systems to minimize this impact for the future.

UPDATE 1:45PM PT: we just had another cache server die on us. These were servers that we upgraded before the holiday break. We are working on setting up new cache servers as we think we have isolated what the issue might be. What this means is that the above will reset again once we have the new cache servers set up.

UPDATE 3:30pm PT: we will be rolling an urgent update to our cache pool which means that everything above will have been reset again, we will be rolling this update in about 15-min, we have downgraded to a version that we were using pre-holidays which should remove these cache server failures we have been experiencing with the upgraded versions. We apologize for the issues and we are working our best to remove these issues as quickly as possible.

UPDATE 4:00pm PT: the new cache pool has been added and we hope that this is the last time that the data above will be flushed. We will be looking immediately to get a project started to move rival history out of cache to a persistent storage so that when our caching servers do go down data loss will be minimal.

 

i noticed it first hand. thank god it didn't last that long, only 1/2 an hour.

 

Yes we were notified right away. We may have to set up some new cache servers in the short term to try and make these failures not happen which would reset those above. In the longer term we will be looking to remove some of the data like feeds and rivals list to a persistent data storage.

 

Seems like there are still some issues. I can't attack my own boss and my Elite Chieftain Call immediately expired.

 

UPDATE 1:45PM PT: we just had another cache server die on us. These were servers that we upgraded before the holiday break. We are working on setting up new cache servers as we think we have isolated what the issue might be. What this means is that the above will reset again once we have the new cache servers set up.

 

UPDATE 3:30pm PT: New cache pool has been rolled out. We will be monitoring the system throughout the day.

 

UPDATE 4:00pm PT: the new cache pool has been added and we hope that this is the last time that the data above will be flushed. We will be looking immediately to get a project started to move rival history out of cache to a persistent storage so that when our caching servers do go down data loss will be minimal.

 

No f.......... Shit i had just send out a level 2 helreginn and the f........ Limit was gone wtf

 

Yeah WTF...guess Dunky's world is gonna come to a screeching halt....Movie Time!