Novērot MI aģentus

MI rāpuļi — no GPTBot līdz ClaudeBot un PerplexityBot — lasa jūsu vietni katru dienu, un lielākā daļa vietņu īpašnieku to nekad neredz. Novērošanas režīms ir SilentShield MI aģentu kontroles 1. līmenis: tas reģistrē, kuri MI pakalpojumi apmeklē kuras lapas — neko nebloķējot un nemainot jūsu vietnes darbību.

Redzamība ir priekšnoteikums kontrolei: tikai tad, kad zināt, kas lasa jūsu saturu, varat saprātīgi izlemt, ko atļaut, ierobežot vai bloķēt. Tāpēc katrs novērojums tiek iedalīts vienā no trim kategorijām:

Apmācība
Rāpulis vāc jūsu saturu, lai apmācītu nākotnes MI modeļus (piem., GPTBot, ClaudeBot, CCBot).
MI meklēšana
Pakalpojums lasa lapas, lai atbildētu uz lietotāju jautājumiem, un var citēt jūsu vietni kā avotu (piem., OAI-SearchBot, PerplexityBot).
Aģenti
Bots reāllaikā darbojas konkrēta lietotāja uzdevumā, piemēram, izpēta produktu vai aizpilda veidlapu jūsu vietnē (piem., ChatGPT-User, Claude-User).

Iestatīšana pēc platformas

WordPress

Iebūvētais observer pienāks ar gaidāmo SilentShield spraudņa atjauninājumu. Līdz tam aktivizējiet novērošanu ar mazu must-use spraudni — viens fails, bez konfigurācijas:

Izveidojiet failu wp-content/mu-plugins/silentshield-observer.php ar šo saturu (mu-spraudņi ielādējas automātiski; lai atspējotu, izdzēsiet failu):

wp-config.phpphp
<?php
/**
 * Plugin Name: SilentShield — AI Agent Observer
 * Description: Reports AI crawler visits to SilentShield (fire-and-forget;
 * IP/UA are hashed server-side, GDPR). Delete this file to disable.
 *
 * Install: save this file as wp-content/mu-plugins/silentshield-observer.php
 * (mu-plugins load automatically — no activation needed). To force it off
 * without deleting: define('SILENTSHIELD_OBSERVER', false) in wp-config.php.
 * An upcoming SilentShield plugin update will build this in.
 */
if (!defined('ABSPATH')) exit;
if (defined('SILENTSHIELD_OBSERVER') && !SILENTSHIELD_OBSERVER) return;

const SS_OBSERVE_URL = "https://api.silentshield.io/api/v1/agent/telemetry";
const SS_API_KEY = "YOUR_API_KEY"; // publishable site key
// For the always-fresh list, cache https://api.silentshield.io/api/v1/agent/bot-directory daily.
$ss_bots = ["GPTBot","OAI-SearchBot","ChatGPT-User","ClaudeBot","Claude-User",
  "PerplexityBot","Perplexity-User","Bingbot","Applebot","Amazonbot","CCBot",
  "Bytespider","meta-externalagent","Google-Agent"];

register_shutdown_function(function () use ($ss_bots) {
  $ua = $_SERVER["HTTP_USER_AGENT"] ?? "";
  $signed = !empty($_SERVER["HTTP_SIGNATURE"]);
  $isBot = $signed;
  foreach ($ss_bots as $b) { if (stripos($ua, $b) !== false) { $isBot = true; break; } }
  if (!$isBot) return;

  if (function_exists("fastcgi_finish_request")) { fastcgi_finish_request(); }

  wp_remote_post(SS_OBSERVE_URL, [
    "timeout"  => 3,
    "blocking" => false,
    "headers"  => ["Content-Type" => "application/json", "x-api-key" => SS_API_KEY],
    "body"     => wp_json_encode(["sightings" => [[
      "ua"     => $ua,
      "ip"     => $_SERVER["REMOTE_ADDR"] ?? "",
      "path"   => strtok($_SERVER["REQUEST_URI"] ?? "/", "?"),
      "method" => $_SERVER["REQUEST_METHOD"] ?? "GET",
    ]]]),
  ]);
});

Next.js

Patstāvīgs middleware — bez SDK. Novērošana darbojas caur event.waitUntil(), ārpus atbildes ceļa, tāpēc tā nekad nepalēnina jūsu lapas:

middleware.tstypescript
// middleware.ts — self-contained observer, no SDK required.
// Sightings are reported via event.waitUntil(), off the response path,
// so your pages never wait. Best-effort and fail-open by design.

import { NextResponse, type NextRequest } from "next/server";

const OBSERVE_URL = "https://api.silentshield.io/api/v1/agent/telemetry";
const API_KEY = "YOUR_API_KEY"; // publishable site key

// Known AI-agent UA tokens. For the always-fresh list, fetch
// https://api.silentshield.io/api/v1/agent/bot-directory once a day and cache it.
const BOTS = ["GPTBot", "OAI-SearchBot", "ChatGPT-User", "ClaudeBot", "Claude-User",
  "PerplexityBot", "Perplexity-User", "Bingbot", "Applebot", "Amazonbot", "CCBot",
  "Bytespider", "meta-externalagent", "Google-Agent"];

export function middleware(req: NextRequest, event: { waitUntil(p: Promise<unknown>): void }) {
  const ua = req.headers.get("user-agent") ?? "";
  const signed = !!req.headers.get("signature");
  if (signed || BOTS.some((b) => ua.toLowerCase().includes(b.toLowerCase()))) {
    const sighting = {
      ua,
      path: req.nextUrl.pathname,
      method: req.method,
      ip: req.headers.get("x-forwarded-for")?.split(",")[0]?.trim(),
      ...(signed && {
        signature: req.headers.get("signature"),
        signature_input: req.headers.get("signature-input"),
        signature_agent: req.headers.get("signature-agent"),
        authority: req.nextUrl.host,
        scheme: req.nextUrl.protocol.replace(":", ""),
      }),
    };
    event.waitUntil(
      fetch(OBSERVE_URL, {
        method: "POST",
        headers: { "Content-Type": "application/json", "x-api-key": API_KEY },
        body: JSON.stringify({ sightings: [sighting] }),
      }).catch(() => {}) // best-effort, fail-open
    );
  }
  return NextResponse.next();
}

Node.js (Express)

Express (vai jebkuram citam Node serverim) pievienojiet fire-and-forget middleware. Tā ziņo tikai par pieprasījumiem, kuru User-Agent atbilst zināmam MI aģentam vai kuriem ir Web Bot Auth paraksts — un nekad nebloķē atbildi:

server.jsjavascript
import express from "express";
const app = express();

const OBSERVE_URL = "https://api.silentshield.io/api/v1/agent/telemetry";
const API_KEY = "YOUR_API_KEY";

// Known AI-agent UA tokens. For the always-fresh list, fetch
// https://api.silentshield.io/api/v1/agent/bot-directory once a day and cache it.
const BOTS = ["GPTBot", "OAI-SearchBot", "ChatGPT-User", "ClaudeBot", "Claude-User",
  "PerplexityBot", "Perplexity-User", "Bingbot", "Applebot", "Amazonbot", "CCBot",
  "Bytespider", "meta-externalagent", "Google-Agent"];

// Fire-and-forget observer — never blocks the response.
app.use((req, res, next) => {
  const ua = req.headers["user-agent"] || "";
  const signed = !!req.headers["signature"];
  if (signed || BOTS.some((b) => ua.toLowerCase().includes(b.toLowerCase()))) {
    const sighting = {
      ua, ip: req.ip, path: req.path, method: req.method,
      ...(signed && {
        signature: req.headers["signature"],
        signature_input: req.headers["signature-input"],
        signature_agent: req.headers["signature-agent"],
        authority: req.headers["host"], scheme: req.protocol,
      }),
    };
    fetch(OBSERVE_URL, {
      method: "POST",
      headers: { "Content-Type": "application/json", "x-api-key": API_KEY },
      body: JSON.stringify({ sightings: [sighting] }),
    }).catch(() => {}); // best-effort, fail-open
  }
  next();
});

PHP

Vienkāršam PHP iekļaujiet šo novērotāju sava front controller pašā sākumā. Tas nosūta novērojumu tikai pēc tam, kad atbilde jau ir piegādāta (fastcgi_finish_request), tāpēc lapa nekad netiek palēnināta:

observer.phpphp
<?php
// Drop-in observer: include at the very top of your front controller.
// It runs after the response is sent (fastcgi_finish_request) so it never
// slows down the page. Reuses the same publishable key.

const SS_OBSERVE_URL = "https://api.silentshield.io/api/v1/agent/telemetry";
const SS_API_KEY = "YOUR_API_KEY";
// For the always-fresh list, cache https://api.silentshield.io/api/v1/agent/bot-directory daily.
$ss_bots = ["GPTBot","OAI-SearchBot","ChatGPT-User","ClaudeBot","Claude-User",
  "PerplexityBot","Perplexity-User","Bingbot","Applebot","Amazonbot","CCBot",
  "Bytespider","meta-externalagent","Google-Agent"];

register_shutdown_function(function () use ($ss_bots) {
  $ua = $_SERVER["HTTP_USER_AGENT"] ?? "";
  $signed = !empty($_SERVER["HTTP_SIGNATURE"]);
  $isBot = $signed;
  foreach ($ss_bots as $b) { if (stripos($ua, $b) !== false) { $isBot = true; break; } }
  if (!$isBot) return;

  if (function_exists("fastcgi_finish_request")) { fastcgi_finish_request(); }

  $sighting = [
    "ua" => $ua,
    "ip" => $_SERVER["REMOTE_ADDR"] ?? "",
    "path" => strtok($_SERVER["REQUEST_URI"] ?? "/", "?"),
    "method" => $_SERVER["REQUEST_METHOD"] ?? "GET",
  ];
  $ch = curl_init(SS_OBSERVE_URL);
  curl_setopt_array($ch, [
    CURLOPT_POST => true,
    CURLOPT_RETURNTRANSFER => true,
    CURLOPT_TIMEOUT => 3,
    CURLOPT_HTTPHEADER => ["Content-Type: application/json", "x-api-key: " . SS_API_KEY],
    CURLOPT_POSTFIELDS => json_encode(["sightings" => [$sighting]]),
  ]);
  curl_exec($ch); // best-effort, fail-open
  curl_close($ch);
});

Go

Patstāvīgs observer dažās rindās — tikai standarta bibliotēka, bez SDK. Novērojumi tiek sūtīti fire-and-forget veidā, tāpēc neviens pieprasījums nekļūst lēnāks:

main.gogo
// Self-contained observer — stdlib only, no SDK required.
// Reports AI-agent sightings fire-and-forget: never blocks a request,
// fail-open on any error. For the always-fresh bot list, cache
// https://api.silentshield.io/api/v1/agent/bot-directory daily.

const observeURL = "https://api.silentshield.io/api/v1/agent/telemetry"
const siteKey = "YOUR_API_KEY" // publishable — or os.Getenv("SILENTSHIELD_SITE_KEY")

var bots = []string{"GPTBot", "OAI-SearchBot", "ChatGPT-User", "ClaudeBot",
    "Claude-User", "PerplexityBot", "Perplexity-User", "Bingbot", "Applebot",
    "Amazonbot", "CCBot", "Bytespider", "meta-externalagent", "Google-Agent"}

func observe(next http.Handler) http.Handler {
    return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
        ua, signed := r.UserAgent(), r.Header.Get("Signature") != ""
        hit := signed
        for _, b := range bots {
            if strings.Contains(strings.ToLower(ua), strings.ToLower(b)) {
                hit = true
                break
            }
        }
        if hit {
            ip, _, _ := net.SplitHostPort(r.RemoteAddr)
            s := map[string]any{"ua": ua, "ip": ip, "path": r.URL.Path, "method": r.Method}
            if signed { // Web Bot Auth (RFC 9421) — forward for verification
                s["signature"] = r.Header.Get("Signature")
                s["signature_input"] = r.Header.Get("Signature-Input")
                s["signature_agent"] = r.Header.Get("Signature-Agent")
                s["authority"] = r.Host
            }
            body, _ := json.Marshal(map[string]any{"sightings": []any{s}})
            go func() { // best-effort, fail-open
                req, _ := http.NewRequest("POST", observeURL, bytes.NewReader(body))
                req.Header.Set("Content-Type", "application/json")
                req.Header.Set("x-api-key", siteKey)
                if resp, err := (&http.Client{Timeout: 3 * time.Second}).Do(req); err == nil {
                    resp.Body.Close()
                }
            }()
        }
        next.ServeHTTP(w, r)
    })
}

// net/http:  http.ListenAndServe(":8080", observe(mux))
// Gin:       srv := &http.Server{Addr: ":8080", Handler: observe(r)}
//            srv.ListenAndServe()

Pārbaudiet savienojumu

Jums nav jāgaida pirmais īstais MI rāpulis, lai pārliecinātos, ka integrācija darbojas. Nosūtiet tukšu paketi — tā reģistrē heartbeat jūsu atslēgai, neierakstot nekādus viltus novērojumus:

Terminalbash
curl -X POST https://api.silentshield.io/api/v1/agent/telemetry \
  -H "Content-Type: application/json" \
  -H "x-api-key: YOUR_API_KEY" \
  --data '{"sightings":[]}'

Jūsu API atslēgas aģentu centrs (API atslēgas → izvēlieties atslēgu → Aģents) tad pārslēdzas no „nav savienots” uz „savienots — klausās”. Jūsu pārskats paliek tukšs, līdz ierodas īsts aģents: SilentShield nekad neizdomā datus.

VDAR

Novērošana balstās uz leģitīmām interesēm (Art. 6(1)(f) VDAR). IP adreses servera pusē tiek hešotas ar sāls vērtību, kas mainās katru dienu — neapstrādātas IP adreses nekad netiek glabātas. Novērojumi tiek glabāti 14 dienas, un vispār tiek saglabāti tikai zināmu MI aģentu pieprasījumi: parasta apmeklētāju datplūsma nekad netiek reģistrēta.

BUJ

Vai tas palēnina manu vietni?

Nē. Katra integrācija ir fire-and-forget: Next.js izmanto event.waitUntil(), PHP ziņo tikai pēc atbildes nosūtīšanas, Node un Go sūta fonā. Un, ja SilentShield API kādreiz nav sasniedzama, jūsu vietne netiek ietekmēta — viss darbojas pēc fail-open principa.

Kas tiek uzskatīts par MI aģentu?

Tikai pieprasījumi, kas atbilst SilentShield zināmo MI rāpuļu un aģentu botu direktorijam (GPTBot, ClaudeBot, PerplexityBot, …) vai kuriem ir Web Bot Auth paraksts. Vienmēr aktuālais saraksts ir pieejams vietnē https://api.silentshield.io/api/v1/agent/bot-directory.

Kāpēc es vēl neredzu datus?

MI rāpuļi ierodas pēc sava grafika. Lielākajā daļā vietņu pirmie novērojumi parādās 24–72 stundu laikā; mazākām vietnēm tas var prasīt ilgāku laiku. Pa to laiku heartbeat jūsu aģentu centrā apliecina, ka pati integrācija darbojas.

Tālāk: piemērot aģentu noteikumus (2. līmenis) →