This week has been a pretty fun week for me, I’ve solved some major problems and created some fantastic things.
Firstly, if you’re a Duxter user, you will know that we have recently been having outbreaks of epic slowness and sometimes the site not loading at all. Initially when this started happening we concluded that it had something to do with Node (our websockets type server) and its communications with our database server. what we had done at this point was started pooling the database connections in node hoping that it would improve things, and it did, slightly, and for a short time. the improvements we seen were simply false indicators of us finding the solution, when in fact the improvements in performance we seen was simply down to the fact it was late at night (for me) and activity was low on the site.
It soon became obvious that our previous fix added some slight improvements but didn’t solve our overloading issues. It soon became a real problem as we were having to reboot the node daemon almost every hour. As all our developers were currently knee deep in other projects, rebooting node because a sort of temporary solution until we had time to look deeper into it… that was until one morning I was trying to work and i was having to restart node every 5 minutes. Now, as almost all of our team is based in the United States (and me being from the United Kingdom) I was all alone and had no choice but to investigate this problem myself. After a bit of stress and some Sherlock Holmes style investigation, I finally found the problem. What made this difficult to track was the fact certain services are hosted on different servers, so even though our production server was showing as being healthy and under no stress at all, the site was taking a really long time to load. So after ruling out our production server was having problems I decided to look elsewhere, Node debug logs shown no issues, and no database errors were produced. At this point I started looking the active connections and CPU usage of our database server. Bingo! there it was, 99.8% CPU usage and an average of about 150 connections at any given time. Unfortunately that was all the information I could obtain from there so I dived straight into the database and ran the “SHOW PROCCESSLIST” query. And right there, the same query running hundreds of times. So after a bit more tracking I’d located it to being an issue with our Instant Messenger system, specifically the friends list functions which run through node. Disabling the IM system instantly reduced the Database CPU usage to nearly nothing, and the active connections dropped to about 5 average. At this moment in time the site and servers are all running fantastically well with not so much as a hiccup. We’ve now located the piece of bad code that caused this, and now have one of our developers making several improvements to the IM systems. So, with that all diagnosed we shoudl have a new version of our IM system running shortly, which I can’t wait for.
So, what have I been working on this week? Well for me, one of our most exciting features, Our Developer Platform. This is the place where third parties can create and publish apps both internally on Duxter for Duxter users to use, externally on their own websites and in native applications for the likes of Windows, Mac OSX and mobile devices. I wont go into to much details about how this is all going to work for end-users and developers at the moment as we’ll be doing a public announcement when we’re ready to launch it. This for me however has been one of my favorite projects for the simple fact creating frameworks and laying down the foundations for such a system is a real challenge. I thrive on achieving the impossible, tackling difficult challenges and pushing my abilities to the absolute max. It’s a great change from what I’ve been working on lately, such as creating “features” for Duxter, whilst I do enjoy that stuff, I find it a bit trivial. I mean, within a few minutes of being proposed an idea of feature, i can see exactly how it is laid out, how it should work and what needs to be done, it just becomes a case of writing the code and tackling any bugs that show up, a perfect example of this is our “Rewards Store” (which I’ll get to in a moment). Creating a system from scratch though, now that takes a bit more thought and with every new class created there’s that awesome sense of accomplishment.
So, the Rewards store. You may have noticed we finally launched our rewards store on Duxter, and this system has been a real pain for us. Initially when we lay down the plans for how things should work on Duxter, and we decided that we should use a pre-built eCommerce system to save us time and allow developers to spend more time on other features. This would require a few things, to cut the story short, using an SSO (single sign on) solution caused slight overhead, we were having to use API’s in order to use user information, and the eCommerce system required a bunch of modding to get it to do what we needed. Seriously, as great as this system was stand-alone, integration was an absolute nightmare. We dropped it last week and said farewell to it for good. After a short meeting with Adam we had a list of primary features of what our Reward Store should do. We laid down a plan, pulled in Robert and Ross (2 of our developers) to start working on certain features and within 2 days we had a fully functional eCommerce system ready to go live. Whilst it’s still in it’s early stages and we have a bunch of things to still add, completing this within 2 days was a real achievement and really shown off each of our developers talents, in fact, it took longer for us to populate the store with products than it did to create the system.
On the business side of things, our CEO Adam has been real busy this week (seriously, the guy is never off the phone). We’ve acquired a second office in Seattle, which will give our CTO Sky a new battleground for his Nerf gun antics (the guy is obsessed with Nerf guns). Adam has also been real busy meeting new start-ups and creating relationships with these companies, and existing companies, but as geeky as I may be, all this business stuff is way over my head so check out his blog if your more interested in the business side of things, he doesn’t get much time to blog about this sort of stuff, but he does try.
So that’s about all I have to share with you this week, there’s been so much happening in all aspects of Duxter my brain is just overloaded with things to tell you about. If your not already a member on Duxter, get yourself involved at http://duxter.com seriously, if your a gamer, you NEED to be here.
Until next week peeps.