How to deal with bad bots? Follow

If your Section application is being hit by bad bots there are a number of things you can do. 

Identify the Bot

The best tool for doing investigations of bot traffic, and determining if a bot is good or bad, is Kibana. 

Using Kibana you can create a report, which looks at edge traffic, country of origin, IP, and user-agent. If you have a spike in traffic from bots from countries where you do not have customers, hitting URI like login, cart or crawling your product catalog, or harvesting your images, you likely have a bad bot. 

Some bad bot identifiers;

• The country of origin is suspicious.
• A big spike in traffic is originating from a suspicious country.
• There is a strange or unusual user agent associated with the bot.
• The bot is rotating through a number of IPs.
• The bot is hitting login or cart.
• The bot is crawling product catalogs or image libraries.

From this Kibana report, you can get the remote_addr which is the IP of the bot, and the user agent. The IP is redacted for GPDR and so, therefore, you will be blocking an entire range of 256 IP when you use this to block traffic. 

In the Section Console > Application Edge > IP Restrictions, you can the remote_addr to the IP restrictions list. 

Add the bad bot IP to the list via the form in the Section Console

Blocking Requests https://www.section.io/docs/how-tos/environment/blocking-requests/

Advanced Solutions

It is also possible to use Varnish VCL to block by IP, user-agent or country code. Similar solutions can be created in OpenResty, although Varnish is more common. 

Bot Management Modules

Section has a range of third-party partners on the Section platform that are WAF and bot management tools. 

Contact a Section engineer to discuss the growing range of bot management and WAF tools available. 

https://www.section.io/docs/reference/modules/