Witching woes, part I

C'est fantastique. Insane market data volumes.

Composite market data service crashed at 9.35 without an explanation. Primary market data service ran out of memory and crashed at 9.58.

And the day is young.

Witching woes, part II

Slow consumption of composite market data by our services, RV disconnected channels for most active tickers cz their queues got too big. I mean, who needs to know the current composite prices for MSFT in a market like todays anyway. Missing composite data for actively trading items for some time.

And the day is still young.

Witching woes, part III

Composite market data box ran with a load of 96 (!), but only for a little while -- composite service cored (again).

And we still have 30 most volume-intensive minutes to go.

Witching woes, part IV

It's finally over.

Between 15.20 and 16.00 composite service cored 5 times because it couldn't process that much shit market data, bringing the total for the day to 7.

PS. Something tells me we will be doing market data services capacity remediation for a few weeks...