[Django]-Load spike protection for Django Channels

6šŸ‘

āœ…

Iā€™ve received an answer from Andrew Godwin. He doesnā€™t use StackOverflow so Iā€™m posting it here on his behalf.

Hi Jamie,

At the moment Channels has quite limited support for throttling ā€“ it pretty much consists of an adjustable channel size for incoming connections which, when full, will cause the server to return a 503 error. Workers are load-balanced based on availability due to the channels design, so thereā€™s no risk of a worker gaining a larger queue than others.

Providing more advanced DoS or DDoS protection is probably not something we can do within the scope of Channels itself, but Iā€™d like to make sure we provide the appropriate hooks. Were there particular things you think we could implement that would help you write some of the things you need?

(Itā€™s also worth bearing in mind that right now weā€™re changing the worker/consumer layout substantially as part of a major rewrite, which is going to mean different considerations when scaling, so I donā€™t want to give too precise advice just yet)

Andrew

Heā€™s also written about the 2.0 migration in his blog.

0šŸ‘

I am only answering the first question. So basically it is impossible to be 100% protected from ddos attacks, because it always comes down to a battle of resources. If the server-side resources are greater than the attacker-side resources, the server will not go down (there may be slowed performance though) but if not, the server goes down [no reference required]. Why is it not possible to be 100% protected, you may ask. So basically your server ā€œcrashesā€ if people cannot connect to it [https://en.wikipedia.org/wiki/Crash_(computing)#Web_server_crashes ā€” Web server crashes sentence 1.]. So if you try to protect your server by shutting it down for 5 mins every time 10000 connections are made in a second, the ddos succeeded. It ā€œcrashedā€ your server. The only ddos protection that I know of that should work is Cloudfare (https://www.cloudflare.com/lp/ddos-b/?_bt=207028074666&_bk=%2Bddos%20%2Bprotection&_bm=b&_bn=g&gclid=EAIaIQobChMIu5qv4e-Z1QIVlyW9Ch2YGQdiEAAYASAAEgJbQ_D_BwE). It absorbs the impact of the ddos attack with its 10Tbps network backbone. But even it does not offer 100% ddos protection because once its 10Tbps is down, your server will go down too. So, I hope that helped.

šŸ‘¤Evgeny

0šŸ‘

DDoS = Distributed Denial of Service

The ā€˜Distributedā€™ part is the key: you canā€™t know youā€™re being attacked by ā€˜someoneā€™ in particular, because requests come from all over the place.

Your server will only accept a certain number of connections. If the attacker manages to create so many connections that nobody else can connect, youā€™re being DDoSā€™ed.

So, in essence you need to be able to detect that a connection is not legit, or you need to be able to scale up fast to compensate for the limit in number of connections.

Good luck with that!

DDoS protection should really be a service from your cloud provider, at the load balancer level.

Companies like OVH use sophisticated machine learning techniques to detect illegitimate traffic and ban the IPs acting out in quasi-real time.
For you to build such a detection machinery is a huge investment that is probably not worth your time (unless your web site is so critical and will lose millions of $$$ if itā€™s down for a bit)

šŸ‘¤MrE

0šŸ‘

Theres a lot of things you cant to do about DDOS..however there are some neat ā€˜tricksā€™ depending on how much resources you have at your disposal, and how much somebody wants to take you offline.

Are you offering a total public service that requires direct connection to the resource you are trying to protect?

If so, you just going to need to ā€˜soak upā€™ DDOS with the resources you have, by scaling up and outā€¦ or even elasticā€¦ either way itā€™s going to cost you money!

or make it harder for the attacker to consume your resources. There are number of methods to do this.

If you service requires some kind of authentication, then separate your authentication services from the resource you are trying to protect.

Many applications, the authentication and ā€˜serviceā€™ run on the same hardware. thats a DOS waiting to happen.

Only let fully authenticated users access the resources you are trying to protect with dynamic firewall filtering rules. If your authenticated then gate to the resources opens (with a restricted QOS in place) ! If your a well known, long term trusted users, then access the resource at full bore.

Have a way of auditing users resource behaviour (network,memory,cpu) , if you see particular accounts using bizarre amounts, ban them, or impose a limit, finally leading to a firewall drop policy of their traffic.

Work with an ISP that can has systems in place that can drop traffic to your specification at the ISP borderā€¦. OVH are your best bet. an ISP that exposes filter and traffic dropping as an API, i wish they existedā€¦ basically moving you firewall filtering rules to the AS borderā€¦ niiiiice! (fantasy)

It wonā€™t stop DDOS, but will give you a few tools to keep resources wasted a consumption by attackers to a manageable level. DDOS have to turn to your authentication serversā€¦ (possible), or compromise many user accountsā€¦. at already authenticated users will still have access šŸ™‚

If your DDOS are consuming all your ISP bandwidth, thats a harder problem, move to a larger ISP! or move ISPā€™sā€¦ :-). Hide you main resource, allow it to be move dynamically, keep on the move! :-).

Break the problem into piecesā€¦ apply DDOS controls on the smaller pieces. šŸ™‚

Iā€™ve tried a most general answer, but there are a lot a of depends, each DDOS mitigation requires a bit of Skin, not tin approach.. Really you need a anti-ddos ninja on your team. šŸ˜‰

Take a look at distributed protocolsā€¦. DPā€™s maybe the answer for DDOS.

Have fun.

0šŸ‘

Letā€™s apply some analysis to your question. A DDoS is like a DoS but with friends. If you want to avoid DDoS explotation you need minimize DoS possibilities. Thanks capitan obvious.

First thing is to do is make a list with what happens in your system and wich resources are affected:

  • A tcp handshake is performed (SYN_COOKIES are affected)
  • A ssl handshake comes later (entropy, cpu)
  • A connection is made to channel layerā€¦

Then monitorize each resource and try to implement a counter-measure:

  • Protect to SYN_FLOOD configuring your kernel params and firewall
  • Use entropy generators
  • Configure your firewall to limit open/closed connection in short time (easy way to minimize ssl handshakes)
  • ā€¦

Separate your big problem (DDoS) in many simple and easy to correct tasks. Hard part is get a detailed list of steps and resources.

Excuse my poor english.

šŸ‘¤lasizoillo

Leave a comment