Site Problem

[Peter Friedman]

All parts of this site are taking an inordinate time to open.  I almost gave up trying to open a number of threads as it was taking forever for any of them to open!

[Dan Gould]

Back to normal for me.

And strangely, as things were so slow to load, I never got the "server is taking too long to respond" browser error-it would still load.

But now speed is normal.

[Peter Friedman]

Yes, things seem to working fine for me now.

[jlhoots]

Really slow today. Better now.

[Kevin Bresnahan]

I have been talking with Jim Alfredson about this. This forum, like all forums across the web, are under regular attacks by AI bots scraping the site for every post ever made. At the Steve Hoffman forums, they posted about how it is affecting them:

Update to our scraper/bot situation.

Over the past year (it's been almost a year to the date that we first posted this thread!), bot/scraper traffic hitting servers around the Internet has likely increased by an order of magnitude. The heavy traffic comes not from traditional sources like the search engine spiders of Bing, Google, DuckDuckGo, etc. (which was always expected and normal traffic), but from what we are calling an "AI arms race" as Big Tech is scraping up data to feed their LLMs at an alarming rate.

Through Cloudflare, we are able to stop the legitimate AI/LLM scrapers from our site (as if plagiarism and lack of compensation is legitimate ), but scraper bots from other countries are undeterred and find ways to get around the various security rules most of us have in place to try to stop them. We are constantly learning of new patterns to deter them with, but that is often not enough.

One of the more recent developments is that these scrapers from other countries are using residential proxies. There are companies out there that sell VPN service to consumers, selling the idea of privacy. Buried in the fine print is a notice that the company will also use the consumer's connection to access the Internet. There are others that offer SDKs (software development kits) that provide a backdoor to consumer devices, like Bright Data. Here's an article describing how they work:

https://archive.ph/IdEHm

The short version is that datacenters are now able to disguise themselves using residential consumer IP addresses, often through phones/tablets and televisions via apps that consumers download. And that is just one of a handful of situations we are dealing with on the server.

Please be aware that our ongoing war in this AI arms race means that we are constantly trying new security rules to try to mitigate these AI scrapers from creating what is essentially a denial of service attack--their bots hit the forum so fast and so frequently that it denies service to you, our human visitors and members. We don't want that experience for any of our visitors!

If you encounter a verification of some sort, any reasonably modern OS and browser will be able to solve the Cloudflare challenge without any intervention at all. (No, you don't need a "bleeding edge" device, OS, or browser to get in, something we were accused of recently. But browsers are free, and for security reasons alone we always suggest you let your browser auto-update itself to the newest version. And please...don't ever use Internet Explorer. The Internet has moved on.) We also have other upgrades planned in the near future, a requirement to deal with this ongoing issue.

Please be patient as this is an ongoing battle with Big Tech and their greedy means of grabbing any and all content, with zero regard for the intense burden they have placed on all web servers around the Internet. You may encounter hiccups. If you find you cannot access the forum, reach out using our Contact Us link at the bottom of the site. Your message will be brought up with staff and forwarded to the technical department.

Important final note--if you are logged in, the cookie used for your account is also used to "whitelist" your access. In other words, if you are logged in, you bypass Cloudflare completely. (All the more reason to sign up for an account if you are a visitor; we don't mind if you never post a message!)

Thank you for hanging around throughout this mess! We appreciate it!

>>>>>>>>>>>>>

This is Kevin again... note how the SH moderator mentions their use of Cloudflare. We don't have that here & it's not free, so it's unlikely that we will get it anytime soon.

[rostasi]

"AI arms race" 😄

[Kevin Bresnahan]

As a moderator, I think I can actually see the problem the Hoffman moderator is talking about...

At the bottom of the main page, it shows how many users are online. As of right now, there are 1,459 users online. Of these 1,459 users, I would guess that about 15 of them are actual members of the forum. The others have IPs that resolve to locations all over the world, which are obviously being spoofed using VPNs.

These guests are checking out topics like "BMG Sale Corner", "Swine Flu. Anyone here come down with it?", "NBA 2012" and "Upcoming RVGs for 2004". Those don't seem like topics that any of us would be very interested in these days.  

[Niko]

clearly off-topic but one of the funnier aspects of those recent events around swarms of AI agents hacking different places is that, evidently, AI has discovered internet message boards as an amazing way of communicating, here is a summary of events around the "hugging face attack" that was coordinated from a message board on which AIs exchanged 70000 messages over the course of a few days

https://www.dwarkesh.com/p/openai-huggingface

while humans have largely given up on message boards... with a few exceptions like us... 

Edited September 10 by Niko

[Dub Modal]

This is terrible news. Social media sites make for awful forums, reddit included. 

[Jim Alfredson]

Hi friends,

I have signed up for Cloudflare and changed the nameservers, which will take up to 24 hours to fully propagate. During this time some problems may arise. Just let me know and I'll try to fix them.

The added cost is another $28 per month. I have been paying for this forum out of pocket for quite some time and it's now up to almost $200 per month. I think the best solution to this problem is to start allowing ads on the site. We have 20,000 unique visitors every month. I'm missing out on thousands of dollars of income per month. I will keep the ads to a minimum but I think that's the best way forward at this point because I simply cannot afford to keep this thing going otherwise.

Thanks for your understanding.

[Kevin Bresnahan]

I have zero problems with ads here Jim. Anything to alleviate the financial burden is fine with me.

I visit many websites that bombard me with ads and I've developed the ability to mostly ignore them by now. The one website I would avoid duplicating is George Takei's percolately.com, which loads so many ads when you visit the site that it often locks up Firefox.

[HutchFan]

12 minutes ago, Kevin Bresnahan said:

I have zero problems with ads here Jim. Anything to alleviate the financial burden is fine with me.

Same here.

 

[Chuck Nessa]

I might buy some.

 

[Jim Alfredson]

So sorry for the long outage. I've been working with the server company to get everything back and running smoothly since 6pm today (currently almost midnight). I think we can all agree that site is snappier than it's been in years. Keeping the bots away should help a lot.