minus-squareAwesomeLowlander@sh.itjust.workstoTechnology@lemmy.world•AI haters build tarpits to trap and trick AI scrapers that ignore robots.txtlinkfedilinkEnglisharrow-up1·1 year agoThe internet being what it is, I’d be more surprised if there wasn’t already a website set up somewhere with a malicious robots.txt file to screw over ANY crawler regardless of providence. linkfedilink
The internet being what it is, I’d be more surprised if there wasn’t already a website set up somewhere with a malicious robots.txt file to screw over ANY crawler regardless of providence.