nytimes.com

9 agent(s) blocked: GPTBot, ChatGPT-User, OAI-SearchBot, PerplexityBot, ClaudeBot, anthropic-ai, cohere-ai, Google-Extended, Applebot-Extended.

Fix · WordPress: Use Yoast/Rank Math’s robots.txt editor (or a plugin) and ensure no Disallow for AI user-agents.

No reachable sitemap.xml. Publish one and reference it in robots.txt.

Fix · WordPress: Yoast/Rank Math generate /sitemap_index.xml automatically. Enable it and list it in robots.txt.

Found /llms.txt but missing H1 header and markdown links.

Fix · WordPress: Upload an llms.txt to your site root (cPanel/SFTP) or via a “raw file” plugin. Build it at /tools/llms-txt-generator.

No HTTP Link: response headers. Emit canonical/alternate/describedby relations so HEAD-only or stream-rendering agents get them without parsing HTML.

Fix: Emit `Link:` headers for canonical, alternate-language, and describedby relations. Agents that fetch HEAD-only or stream-render rely on these.

Visible text is 1.3% of HTML weight (15933/1264887 bytes). Low. Markup may be crowding out agent-usable text.

Fix: Heavy ad or navigation markup crowds out agent-usable text. Aim for a text-to-DOM ratio above 15%.

Browser pass unavailable; HTML looks like a JS-shell (e.g. #__next / #root). Likely JS-dependent.

Fix · WordPress: Most themes render server-side. Check that page builders (Elementor/Divi) aren’t deferring text to JavaScript, and disable JS lazy-loading for body copy.

# Allow AI agents (AgentSpeed-generated)
User-agent: GPTBot
Allow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: ChatGPT-User
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Google-Extended
Allow: /

Sitemap: https://nytimes.com/sitemap.xml

The New York Times - Breaking News, US News, World News and Videos Skip to content Skip to site index U.S. International Canada Español 中文 Today’s Paper Ir Home U.S. SECTIONS U.S. Politics New York California Education Health Obituaries Science Climate Weather Sports Business Tech The Upshot The Magazine TOP STORIES Donald Trump Supreme Court Congress Immigration Abortion NEWSLETTERS The Morning Make sense of the day’s news and ideas. The Evening Catch up on big news, and wind down to end your day. See all newsletters PODCASTS The Daily The biggest stories of our time, in 20 minutes a day. See all podcasts World SECTIONS World Africa Americas Asia Australia Canada Europe Middle East Science Climate Weather Health Obituaries TOP STORIES Middle East Crisis Russia-Ukraine War China International Relations The Global Profile Leer en Español NEWSLETTERS The World A guide to understanding the news, made for readers around the globe. The Interpreter Original analysis on the week’s biggest global stories. Your Places: Global Update The latest news for any part of the world you select. Canada Letter Backstories and analysis from our Canadian correspondents. See all newsletters Business SECTIONS Business Tech Economy Media Finance and Markets DealBook Personal Tech Energy Transition Your Money TOP STORIES U.S. Economy Stock Market Artificial Intelligence NEWSLETTERS DealBook The most crucial business and policy news you need to know. On Tech Our best tech reporting from the week. See all newsletters PODCASTS Hard Fork Our tech journalists help you make sense of the rapidly changing tech world. See all podcasts Arts SECTIONS Today's Arts Book Review Best Sellers Dance Movies Music Television Theater Pop Culture T Magazine Visual Arts RECOMMENDATIONS 100 Best Movies of the 21st Century Critic’s Picks What to Read What to Watch What to Listen To 5 Minutes to Make You Love Music NEWSLETTERS Books Reviews, news and features to help you find your next read. Watching Streaming TV and movie recommendations. See all newsletters PODCASTS Book Review The podcast that takes you inside the literary world. Popcast Pop music news, new songs and albums, and artists of note. See all podcasts Lifestyle SECTIONS All Lifestyle Well Travel Style Real Estate Food Love Your Money Personal Tech T Magazine COLUMNS 36 Hours Ask Well The Hunt Modern Love Where to Eat Vows Social Q’s The Ethicist Ask the Therapist NEWSLETTERS The Weekender Unwind with stories about culture and the way we live. Well Essential news and guidance to live your healthiest life. See all newsletters PODCASTS Modern Love The complicated love lives of real people. See all podcasts Opinion SECTIONS Opinion Guest Essays Editorials Op-Docs Videos Letters TOPICS Politics World Business Tech Climate Health Culture COLUMNISTS Jamelle Bouie Ross Douthat Maureen Dowd David French Thomas L. Friedman M. Gessen Michelle Goldberg Ezra Klein Nicholas Kristof Carlos Lozada Tressie McMillan Cottom Lydia Polgreen Bret Stephens PODCASTS Interesting Times with Ross Douthat Mapping the new world order through interviews and conversations. The Opinions You've heard the news, here's what to make of it. The Ezra Klein Show Discussions of ideas that matter, plus book recommendations. See all podcasts Video Playlists Today's Videos U.S. Politics Immigration NY Region Science Business Culture Books Wellness World Africa Americas Asia South Asia Donald Trump Middle East Crisis Russia-Ukraine Crisis Visual Investigations Opinion Video See all videos Audio Listen The Headlines The Daily Hard Fork The Ezra Klein Show Interesting Times The Opinions Serial Productions The Book Review Podcast Modern Love Popcast Reporter Reads Cannonball The Interview See all audio Featured Cannonball with Wesley Morris Come on in, the culture's fine. The Headlines Your morning listen. Top stories, in 10 minutes. Serial: The Idiot Compassion has its limits when it comes to your own cousin. Newsletters Audio Our editors share their favorite listens from The Times and beyond. Serial Be the first to know about our new shows. See all newsletters Games Word games, logic puzzles and crosswords, including an extensive archive. Go to Games Play Crossplay NEW Wordle Spelling Bee The Crossword The Midi Crossword NEW The Mini Crossword Connections Strands Pips Sudoku Tiles Letter Boxed Community Spelling Bee Forum Wordplay Column Wordle Review Submit a Crossword Meet Our Crossword Constructors Mini to Maestro Wordlebot Take the Puzzle Personality Quiz Newsletters Gameplay Puzzles, brain teasers, solving tips and more. Easy Mode Get an easy version of one of the hardest crossword puzzles of the week. See all newsletters Cooking Recipes, advice and inspiration for everyday cooking, special occasions and more. Go to Cooking Recipes Easy Dinner Quick Healthy Breakfast Vegetarian Vegan Chicken Pasta Dessert Editors' Picks Easy Salmon Recipes Grilling Recipes Easy Weeknight Newest Recipes Cheap and Easy Dinner Ideas Slow Cooker Recipes Healthy Breakfast Ideas Newsletters The Cooking Newsletter New recipes, easy dinner ideas and smart kitchen tips from Melissa Clark, Sam Sifton and our New York Times Cooking editors. The Veggie Delicious vegetarian recipes and tips from Tanya Sichynsky. Five Weeknight Dishes Dinner ideas for busy people from Emily Weinstein. See all newsletters Wirecutter Reviews and recommendations for thousands of products. Go to Wirecutter Reviews Kitchen Tech Sleep Appliances Home and Garden Moving Travel Gifts Deals Baby and Kid Health and Fitness The Best... Air Purifier Electric Toothbrush Pressure Washer Cordless Stick Vacuum Office Chair Robot Vacuum Newsletters The Recommendation The best independent reviews, expert advice and intensively researched deals. Clean Everything Step-by-step advice on how to keep everything in your home squeaky clean. See all newsletters The Athletic Personalized coverage of your sports teams and leagues. Go to The Athletic LEAGUES N.F.L. N.B.A. N.H.L. Premier League M.L.B. College Football N.C.A.A. Men’s N.C.A.A. Women’s Tennis F1 Fantasy & Betting W.N.B.A. TOP STORIES Today's Headlines 2026 Men’s World Cup NEWSLETTERS The Pulse Delivering the top stories in sports, Sunday to Friday. World Cup Briefing Daily updates and insights from the 2026 men's World Cup. See all newsletters PLAY Connections: Sports Edition Connections: Soccer Edition Connections Coach U.S. International Canada Español 中文 New York Times - Top Stories Prime Minister Andy Burnham and his wife, Marie-France van Heel, at 10 Downing Street in London on Monday. Dan Kitwood/Getty Images Burnham hugs his mother on his first day in office. Kevin Coombs/Reuters Burnham chopping mushrooms during a visit to the Passage Homeless Charity in Victoria, London. Pool photo by Christopher Furlong At Buckingham Palace, King Charles III of Britain invited Burnham to form a government. Pool photo by Aaron Chown Crowds watch as Keir Starmer departs No. 10 Downing Street after resigning. Temilade Adelaja/Reuters Starmer delivers a farewell speech in which he offers his full support to Mr. Burnham. Dan Kitwood/Getty Images Starmer embraces his wife outside No. 10 Downing Street. Temilade Adelaja/Reuters Day 1 as Prime Minister: A Homeless Shelter, a Palace and Downing Street As Andy Burnham became Britain’s seventh leader in a decade, he tried to show that he would govern differently. 5 min read Burnham Makes Surprise Pick for Top Finance Job 4 min read Advertisement SKIP ADVERTISEMENT War in the Middle East U.S. Casualties Risk of Wider War Diplomacy Efforts Red Sea Blockade Strait of Hormuz Analysis As Trump Sidelines Congress, G.O.P. Is Boxed In on Iran War Republicans now face a worst-case political scenario of being asked to fund an unpopular war over which they have exercised little oversight. 5 min read Mohammed Huwais/Agence France-Presse — Getty Images LIVE July 21, 2026, 2:24 a.m. ET Iran-Backed Houthis Expand Conflict With

Running AI agents of your own? AgentSpeed also monitors them in production — every run, its latency, cost, and failures, with alerts when something breaks. Free for 10,000 events a month, no credit card.