Meta Faces Global Backlash Over Website Scraping and Robots.txt Policies

Meta is facing growing criticism from publishers, media organizations and legal experts over policies that could allow the company to scrape website content even when publishers attempt to block automated access through robots.txt.

Experts question whether Meta will honor robots.txt instructions

Benjamin Sobel, a law professor at the University of Wisconsin, told the Nieman Institute that Meta should add specific language to its policy confirming that the company will honor scraping instructions in robots.txt files.

Compliance with robots.txt is voluntary, however. Experts suggest that Meta may instead rely on enforceable terms of service that do not reference robots.txt, potentially allowing it to scrape websites that attempt to block automated access.

“When it comes to robots.txt, you can express your preferences, but unless it’s legally enforceable, you can still ignore it,” Sobel said.

Meta’s scraping policy draws international criticism

Criticism extends beyond the Nieman Institute’s coverage, with media organizations around the world urging regulators to intervene and prevent Meta from imposing what they describe as controversial conditions.

Theo Bamber, chief executive of the British News Media Association, accused Meta of a “shameless land grab” based on “arrogant technology platform bullying of the worst kind.”

Bamber called on UK regulators to “stand up for British creators and the business community” by “stopping Meta from setting this terrible precedent.”

Meanwhile, the European Publishers Council has already asked the European Commission to intervene.

“Publishers must be able to decide whether and how their journalism is used and negotiate fair payment,” said Angela Mills Wad, the group’s executive director. “Summary of such broad claims in platform language is no substitute for consent.”

Similarly, Daniel Coffey, president and CEO of the News/Media Alliance—which the Nieman Institute notes is the largest news publishing industry association in the United States—protested that the condition was “extraordinary and possibly illegal.”

US media group calls Meta’s policy an abuse of market power

In the United States, Coffey called on regulators to recognize that Meta’s “decision to withhold access to viewers unless companies allow Meta to collect content from their websites is an astonishing and dishonest policy and a clear abuse of the company’s market power.”

“We cannot allow it to go into effect on October 30th,” Coffey said.

Source: arstechnica.com