AI Crawler Access for Manufacturers: Public Products, Private Files

Give public product information an entrance and confidential files a boundary

A manufacturer can allow search discovery while making a separate decision about model training. Start by classifying the information, then choose the relevant crawler controls, and finally verify what the production server actually delivers. A permissive robots.txt file is not proof of a successful visit, and a restrictive one is not protection for a confidential drawing.

This guide is for teams responsible for a public catalogue, technical downloads and a customer portal. It proposes an acceptance procedure, not a promise that an AI service will mention your company. It also does not require allowing every automated visitor or weakening the security of your website.

The practical deliverable is a small access register: each representative URL, its intended audience, the permitted use, the observed response and the person responsible for corrections. That register gives marketing, engineering and the website supplier something concrete to agree on.

Why the decision needs more than a single block-AI switch

Cloudflare announced a Disallow AI Training setting on 15 September 2026, addressing the difficulty of crawlers that serve more than one purpose. Its announcement is a reason to review an existing policy, not evidence that a particular website has that configuration enabled. Account settings, other firewall rules and the operator's current commitments still need checking. See the Cloudflare announcement in References.

For a manufacturer, discovery and disclosure are different business decisions. A public pump specification helps a buyer evaluate suitability. A customer-specific assembly drawing may contain information that was never approved for public distribution. Neither file should inherit its policy simply because both happen to be PDFs in the same uploads directory.

Separate the controls before changing a rule

OpenAI documents OAI-SearchBot for search discovery and GPTBot for content that may be used in model training. Their settings are independent. ChatGPT-User describes user-initiated access and is not the automatic search crawler; robots rules may not apply to that access. Use the current OpenAI crawler documentation rather than treating every request containing an AI brand as the same activity.

Google distinguishes Googlebot from the Google-Extended product token. Google-Extended governs specified Gemini training and grounding uses; it is not a separate HTTP user-agent and does not determine inclusion or ranking in Google Search. Do not look for a Google-Extended request in the server log as proof that the policy works.

Write down the intended use before approving a setting: public search discovery, model training, or a user-requested page visit. If management has not decided on training, record that decision as pending rather than silently changing it while fixing a search problem.

Build a content register that an engineer can test

Choose a representative sample, not just the homepage. Include a product-family page, one detailed specification, an approved public PDF, a translated product page, the enquiry page and a private portal download. Check the final URL after redirects, because an old download link may resolve to a different file or host.

Public catalogue: the intended result is an accessible description with the approved model, units, operating limits and contact route. Its purpose is evaluation by prospective buyers.

Public technical document: the intended result is the approved revision, with a clear document title and a way to reach the current product page. Confirm that the document contains no customer-specific dimensions or annotations.

Restricted engineering file: the intended result is an authentication or authorization boundary. Knowing the file URL must not be sufficient to retrieve it. Test a logged-out session and an account without permission, using an authorized test file rather than a customer's real confidential drawing.

For US and UK sales teams, include the actual English pages they send to buyers. For a German-language catalogue, include the German page and its linked documents if such a version exists. A working English homepage is not evidence that all language paths or file hosts behave the same way.

Inspect policy, delivery and crawler evidence separately

First, review the effective robots.txt on each relevant host with the developer. Identify crawler-specific groups and the paths they cover. Preserve a copy and the reason for each proposed change. Google explains that robots.txt manages crawling and is not a security mechanism; restricted material needs real access controls.

Second, inspect delivery at the CDN and origin. For a public product URL, verify the final status, content type and actual response body. A 200 response containing a challenge or sign-in page is not a successful product fetch. Look for the approved model number and a distinctive specification sentence in the delivered HTML, and verify that essential downloads resolve correctly.

Third, review genuine crawler requests where available. A user-agent string alone can be imitated. Compare with the provider's published verification guidance or IP information and the relevant CDN evidence. Replaying a user-agent from your laptop can help diagnose a rule, but it does not prove that a provider's production crawler has reached the page.

Keep these findings separate in the register. “Public page works for our test client” and “verified search crawler fetched the page” are different observations. If no genuine request appears, record “not observed”; that does not establish a block.

Use a repeatable acceptance worksheet

For every sampled URL, capture the policy owner, intended access, test time, final status, final URL, whether the expected content was returned and the evidence location. Store sensitive request details privately. A concise record is more useful than an unexplained green score.

Example acceptance case: an approved motor datasheet is public. The product page and PDF both load without authentication, the visible model and revision agree, and the contact link reaches the correct language form. A separate customer drawing remains inaccessible without the required account permission. This is a proposed test case, not a reported client result.

If the product page passes but the PDF is blocked, resolve the document-specific cause. If the private drawing is accessible anonymously, stop its public delivery and review the exposure with the owner. Do not compensate for either problem by turning off the whole firewall.

Roll out one scoped change at a time. Save the previous policy, identify a rollback owner, repeat the public and restricted tests, and compare subsequent verified requests. Leave rate limits and unrelated protections in place unless there is a documented reason to change them.

Measure discovery without confusing it with permission

After technical acceptance, monitor visits, citations you can actually reproduce, and relevant enquiries. Google says its search AI features do not need special optimization files or a special schema type. Eligible, accessible content still has to be useful; access does not guarantee selection, a citation or customer demand.

Keep three dates: the configuration change, the first observed successful crawler visit and the first observed referral or citation. They may differ, and some may remain unknown. Do not label all organic search traffic as AI traffic or infer that a training opt-out caused a ranking movement from a small sample.

Common mistakes to avoid

Do not treat a robots exclusion as a locked door. Do not authorize all requests merely because their user-agent names a trusted provider. Do not assume blocking a training token blocks the same company's search product. Do not use a successful homepage request to sign off multilingual pages, PDFs and a customer portal together.

The useful business outcome is a controlled public product library: buyers and permitted discovery systems can reach approved information, while files requiring a commercial or engineering relationship stay behind an appropriate access boundary. Review the register after a CDN change, a portal release, a domain migration or a new document publishing workflow.

References

• OpenAI — Overview of crawlers ↗
• Google — Common crawlers and Google-Extended ↗
• Google — Introduction to robots.txt ↗
• Google — AI features and your website ↗
• Cloudflare — Search discovery and AI training controls, 15 September 2026 ↗