Although working for ecommerce companies gave me opportunity to always witness the peak planning and feel the challenge of it. But I think this was the first time when I could actually get a lot of hands-on and much more exposure.
Now I have earlier worked for certain companies which where pathetic when it comes to peak planning. There systems will go down every Saturday and Sunday. There database commits will be pathetic in nature and mostly not indexed. There code is pushed even with System.out.println() ; god knows what sort of code review that was and have they even heard of SONAR.
There streaming systems where designed in a way that would kill the very soul and purpose of the streaming systems.
Yet the engineering managers and CTO will take pride of the fact that there engineers are so dedicated and working even on weekends.
But thanks to our technical leadership we are 1000 times better than them .
Below are the few things which I learnt this time:
- Every thing developed right from the design time must be performance oriented . So when there is a design being planned it should not be concentrated only on output but how the output is being achieved. So for example in one of previous orgs the developer working on a payment system used polling rather than a webhook and imagine how the performance has been affected although objective was met.
- Caching mechanisms play a great role. Now many engineers believe that caching and datastructures all this matters only for search based companies and for social networking giants. But that is completely wrong.
There should be stress and evaluation on what sort of cache to be implemented and the caching mechanism to be followed. This helps a lot in enhancing the customer experience. - Streaming systems. Widely used nowadays but becomes pain when implemented in wrong manner. Again in one of the previous orgs the streaming system was something which resulted in double entries for every commission that was suppose to go to a vendor. Now in such cases we should evaluate and plan our streaming mechanism. Whether a need for back -pressure protocol is there or not needs to be evaluated. Even the consumer whether that can perform best with this streaming system must be evaluated.
- Rate limiter. Now rate limiter does not apply only at the load balancer level. But based on business goals and also on diverse geographies where we serve rate limiting needs to be implemented at API level. Thus to some extent cutting down on DDOS threat.
- Doing performance testing in religious manner. Load testing should happen in strict manner . Not from some laptop over an internet connection. It should have proper test bed. There should be proper dashboard to capture all metrics. The IOPS must be considered. The old data must be used to derive the BUSINESS AS USUAL metrics and then implement the same. A performance testing being done in sincere manner yields best of the best results during peak hours.
- Logs as mentioned earlier can be bane if too much is logged. For this logs must be cut down and what needed must be only logged. printf and System.out.println() such things must not be allowed by implementing strict sonar rules. Also SONAR rules must be implemented to check variable declarations are proper or not. If something can be handled using static then that is best way rather than creating new instance every time.
- Implementing certain business rules to cut down on high incoming volume. Definitely cannot comment much on this as this is something internal.
Do let me know your opinions on this and things which you want to share.
Thanks,
Akash Das
Disclaimer : The points listed above are totally my personal view and in no way represent Narvar Inc’s view or inputs
