The stress level around ENS right now is epic.
You'd be amazed about how fast you have to learn when thrown into something. Like a drunken father, tossing his son into the lake w/o a life vest, you learn to swim by instinct and adrenaline more than any conscience method. That describes the vibe right now perfectly: Caffeinated adrenaline on stress steroids.
We've passed content complete, and are chugging towards ZBR full steam ahead, and I just can't seem to get caught up with the clusterfuck of bugs that keep landing on my plate. And not your tiny, "whatever, we can ship with it" bugs; no, these are the kind that carry strap-ons and chant Gregorian hymms while walking down the street glaring at you.
Memory has been a big problem all-of-a sudden. With the push to content complete, we realized that about 4x assets went into the game in about a 48 hour period, and that none of our systems or budgets expected that load (and no one was policing them in the process). As an oddity, we've recently come under the question of 'Can we ask the OS for every block of allocated memory?' Basically, there's a problem when you try to override malloc/new/delete/HeapAlloc/XMemAlloc/XPhysicalAlloc etc. in that a 3rd party library can easily bypass one of those and go right to the hardware to allocate data. To my dismay, going back and forth with XboxDev support provided no insight into the potential to track down allocations that were coming from 3rd party libs that we didn't have tracked in our heaps. On windows, I've been able to do this sort of thing using an external hooking system. A strategy most noted for hackers and keyloggers, this allows me to hook into any app and intercept all calls to it in order to redirect them to my own app. But, XBOX doesn't play as nicely, and I couldn't get that same system up and running (I tried..)
At this point we have to throw our hands up on the lost memory. My estimate is that we've always been missing this large block, and until all the content went in, we never really started getting close to needing it. With more time, we may track it down, but with only a month left until ZBR, there's bigger fish to fry (like the list of 30 heisenbugs on my list right now...
With a 4th quarter QB sneak, I was able to shave us off about 20mb per scenario and give us that memory back across the board, which for now, puts every map in the green, and keeps us from (hopefully) having to make decisions about cutting larger features from the game. We should be able to do a per-asset analysis of textures to get all our maps above the budget needed.
Performance is next on our radar. It's no secret that we've written our rendering engine elements to be multi threaded as much as they can. Taking full advantage of the awesome 360 hardware we've got available. The sim though, limped along, and never made the transition. So now, we've got around a 100ms sim frame that can easily bounce above 120 if you drop a few extra groups of marines on the map.
The problem here, is how do you accurately fix performance without knocking over the house of cards that is the simulation? Multithreading it now seems like a lost cause, We'd introduce more bugs then we'd have time to fix. So we may just have to settle for multiple render frames per sim frame and ship it. I hate the fact we have to do this, but to be honest, I could see it coming.. no one really took the ball on making the sim work 'right' there was just a group of programmers who were constantly focused on making it 'work' at all. The render frame is still around 30ms though ;)
I'm finding out ungodly amounts of tidbits of information about our engine that I never knew.. small pieces of data that would crumple in the light if you brought them to the surface, so you can only behold them in the belly of the evil beast. But I suppose all the stress is forcing me to learn this stuff. Which can't be all that bad, I should have learned it from the beginning, but just didn't have time to play catchup with our tech-lead programmer; I had a full other list of things that had to get done in parallel.
If you haven't read it, I really suggest "Why don't zebra's get ulcers" I've found it an excellent reference to the human stress system, and the effects on the human body. It really makes you take note of how mental stress really does fuck your living world up, and why working 15 hour days for the next 4 months is not going to go well for my health.
But fuck it; it could be worse I suppose. At least I don't have to worry about getting killed by a Lion when I walk out the door in the morning.
*shrug*
~Main
10.23.2008
Subscribe to:
Posts (Atom)