Opazovanje agentov UI

Pajki UI — GPTBot, ClaudeBot, PerplexityBot in drugi — vsak dan berejo vaše spletno mesto, večina upravljavcev pa tega sploh ne vidi. Način opazovanja je 1. stopnja SilentShieldovega nadzora agentov UI: beleži, katere storitve UI obiskujejo katere strani — ne da bi karkoli blokiral in ne da bi spremenil delovanje vašega spletnega mesta.

Vidnost je pred nadzorom: šele ko veste, kdo bere vaše vsebine, lahko smiselno odločite, koga boste dovolili, omejili ali blokirali. Vsako opažanje se zato razvrsti v eno od treh kategorij:

Učenje
Pajek zbira vaše vsebine za učenje prihodnjih modelov UI (npr. GPTBot, ClaudeBot, CCBot).
Iskanje UI
Storitev bere strani, da odgovori na vprašanja uporabnikov, in lahko pri tem vaše spletno mesto navede kot vir (npr. OAI-SearchBot, PerplexityBot).
Agenti
Bot v realnem času deluje v imenu konkretne osebe ter na primer razišče izdelek ali izpolni obrazec na vašem spletnem mestu (npr. ChatGPT-User, Claude-User).

Nastavitev po platformah

WordPress

Vgrajeni observer prihaja s prihajajočo posodobitvijo vtičnika SilentShield. Do takrat spremljanje aktivirate z majhnim must-use vtičnikom — ena datoteka, brez konfiguracije:

Ustvarite datoteko wp-content/mu-plugins/silentshield-observer.php s to vsebino (mu-vtičniki se naložijo samodejno; za izklop datoteko izbrišite):

wp-config.phpphp
<?php
/**
 * Plugin Name: SilentShield — AI Agent Observer
 * Description: Reports AI crawler visits to SilentShield (fire-and-forget;
 * IP/UA are hashed server-side, GDPR). Delete this file to disable.
 *
 * Install: save this file as wp-content/mu-plugins/silentshield-observer.php
 * (mu-plugins load automatically — no activation needed). To force it off
 * without deleting: define('SILENTSHIELD_OBSERVER', false) in wp-config.php.
 * An upcoming SilentShield plugin update will build this in.
 */
if (!defined('ABSPATH')) exit;
if (defined('SILENTSHIELD_OBSERVER') && !SILENTSHIELD_OBSERVER) return;

const SS_OBSERVE_URL = "https://api.silentshield.io/api/v1/agent/telemetry";
const SS_API_KEY = "YOUR_API_KEY"; // publishable site key
// For the always-fresh list, cache https://api.silentshield.io/api/v1/agent/bot-directory daily.
$ss_bots = ["GPTBot","OAI-SearchBot","ChatGPT-User","ClaudeBot","Claude-User",
  "PerplexityBot","Perplexity-User","Bingbot","Applebot","Amazonbot","CCBot",
  "Bytespider","meta-externalagent","Google-Agent"];

register_shutdown_function(function () use ($ss_bots) {
  $ua = $_SERVER["HTTP_USER_AGENT"] ?? "";
  $signed = !empty($_SERVER["HTTP_SIGNATURE"]);
  $isBot = $signed;
  foreach ($ss_bots as $b) { if (stripos($ua, $b) !== false) { $isBot = true; break; } }
  if (!$isBot) return;

  if (function_exists("fastcgi_finish_request")) { fastcgi_finish_request(); }

  wp_remote_post(SS_OBSERVE_URL, [
    "timeout"  => 3,
    "blocking" => false,
    "headers"  => ["Content-Type" => "application/json", "x-api-key" => SS_API_KEY],
    "body"     => wp_json_encode(["sightings" => [[
      "ua"     => $ua,
      "ip"     => $_SERVER["REMOTE_ADDR"] ?? "",
      "path"   => strtok($_SERVER["REQUEST_URI"] ?? "/", "?"),
      "method" => $_SERVER["REQUEST_METHOD"] ?? "GET",
    ]]]),
  ]);
});

Next.js

Samostojen middleware — brez SDK-ja. Spremljanje teče prek event.waitUntil(), zunaj poti odgovora, zato nikoli ne upočasni vaših strani:

middleware.tstypescript
// middleware.ts — self-contained observer, no SDK required.
// Sightings are reported via event.waitUntil(), off the response path,
// so your pages never wait. Best-effort and fail-open by design.

import { NextResponse, type NextRequest } from "next/server";

const OBSERVE_URL = "https://api.silentshield.io/api/v1/agent/telemetry";
const API_KEY = "YOUR_API_KEY"; // publishable site key

// Known AI-agent UA tokens. For the always-fresh list, fetch
// https://api.silentshield.io/api/v1/agent/bot-directory once a day and cache it.
const BOTS = ["GPTBot", "OAI-SearchBot", "ChatGPT-User", "ClaudeBot", "Claude-User",
  "PerplexityBot", "Perplexity-User", "Bingbot", "Applebot", "Amazonbot", "CCBot",
  "Bytespider", "meta-externalagent", "Google-Agent"];

export function middleware(req: NextRequest, event: { waitUntil(p: Promise<unknown>): void }) {
  const ua = req.headers.get("user-agent") ?? "";
  const signed = !!req.headers.get("signature");
  if (signed || BOTS.some((b) => ua.toLowerCase().includes(b.toLowerCase()))) {
    const sighting = {
      ua,
      path: req.nextUrl.pathname,
      method: req.method,
      ip: req.headers.get("x-forwarded-for")?.split(",")[0]?.trim(),
      ...(signed && {
        signature: req.headers.get("signature"),
        signature_input: req.headers.get("signature-input"),
        signature_agent: req.headers.get("signature-agent"),
        authority: req.nextUrl.host,
        scheme: req.nextUrl.protocol.replace(":", ""),
      }),
    };
    event.waitUntil(
      fetch(OBSERVE_URL, {
        method: "POST",
        headers: { "Content-Type": "application/json", "x-api-key": API_KEY },
        body: JSON.stringify({ sightings: [sighting] }),
      }).catch(() => {}) // best-effort, fail-open
    );
  }
  return NextResponse.next();
}

Node.js (Express)

Za Express (ali kateri koli drug strežnik Node) dodajte middleware tipa fire-and-forget. Poroča samo zahteve, katerih User-Agent se ujema z znanim agentom UI ali ki nosijo podpis Web Bot Auth — in odgovora nikoli ne blokira:

server.jsjavascript
import express from "express";
const app = express();

const OBSERVE_URL = "https://api.silentshield.io/api/v1/agent/telemetry";
const API_KEY = "YOUR_API_KEY";

// Known AI-agent UA tokens. For the always-fresh list, fetch
// https://api.silentshield.io/api/v1/agent/bot-directory once a day and cache it.
const BOTS = ["GPTBot", "OAI-SearchBot", "ChatGPT-User", "ClaudeBot", "Claude-User",
  "PerplexityBot", "Perplexity-User", "Bingbot", "Applebot", "Amazonbot", "CCBot",
  "Bytespider", "meta-externalagent", "Google-Agent"];

// Fire-and-forget observer — never blocks the response.
app.use((req, res, next) => {
  const ua = req.headers["user-agent"] || "";
  const signed = !!req.headers["signature"];
  if (signed || BOTS.some((b) => ua.toLowerCase().includes(b.toLowerCase()))) {
    const sighting = {
      ua, ip: req.ip, path: req.path, method: req.method,
      ...(signed && {
        signature: req.headers["signature"],
        signature_input: req.headers["signature-input"],
        signature_agent: req.headers["signature-agent"],
        authority: req.headers["host"], scheme: req.protocol,
      }),
    };
    fetch(OBSERVE_URL, {
      method: "POST",
      headers: { "Content-Type": "application/json", "x-api-key": API_KEY },
      body: JSON.stringify({ sightings: [sighting] }),
    }).catch(() => {}); // best-effort, fail-open
  }
  next();
});

PHP

Za čisti PHP vključite ta opazovalnik čisto na vrh svojega front controllerja. Opažanje pošlje šele po tem, ko je bil odgovor izročen (fastcgi_finish_request) — stran zato nikoli ni počasnejša:

observer.phpphp
<?php
// Drop-in observer: include at the very top of your front controller.
// It runs after the response is sent (fastcgi_finish_request) so it never
// slows down the page. Reuses the same publishable key.

const SS_OBSERVE_URL = "https://api.silentshield.io/api/v1/agent/telemetry";
const SS_API_KEY = "YOUR_API_KEY";
// For the always-fresh list, cache https://api.silentshield.io/api/v1/agent/bot-directory daily.
$ss_bots = ["GPTBot","OAI-SearchBot","ChatGPT-User","ClaudeBot","Claude-User",
  "PerplexityBot","Perplexity-User","Bingbot","Applebot","Amazonbot","CCBot",
  "Bytespider","meta-externalagent","Google-Agent"];

register_shutdown_function(function () use ($ss_bots) {
  $ua = $_SERVER["HTTP_USER_AGENT"] ?? "";
  $signed = !empty($_SERVER["HTTP_SIGNATURE"]);
  $isBot = $signed;
  foreach ($ss_bots as $b) { if (stripos($ua, $b) !== false) { $isBot = true; break; } }
  if (!$isBot) return;

  if (function_exists("fastcgi_finish_request")) { fastcgi_finish_request(); }

  $sighting = [
    "ua" => $ua,
    "ip" => $_SERVER["REMOTE_ADDR"] ?? "",
    "path" => strtok($_SERVER["REQUEST_URI"] ?? "/", "?"),
    "method" => $_SERVER["REQUEST_METHOD"] ?? "GET",
  ];
  $ch = curl_init(SS_OBSERVE_URL);
  curl_setopt_array($ch, [
    CURLOPT_POST => true,
    CURLOPT_RETURNTRANSFER => true,
    CURLOPT_TIMEOUT => 3,
    CURLOPT_HTTPHEADER => ["Content-Type: application/json", "x-api-key: " . SS_API_KEY],
    CURLOPT_POSTFIELDS => json_encode(["sightings" => [$sighting]]),
  ]);
  curl_exec($ch); // best-effort, fail-open
  curl_close($ch);
});

Go

Samostojen observer v nekaj vrsticah — samo standardna knjižnica, brez SDK-ja. Zaznave se pošiljajo fire-and-forget, zato nobena zahteva ne pridobi zakasnitve:

main.gogo
// Self-contained observer — stdlib only, no SDK required.
// Reports AI-agent sightings fire-and-forget: never blocks a request,
// fail-open on any error. For the always-fresh bot list, cache
// https://api.silentshield.io/api/v1/agent/bot-directory daily.

const observeURL = "https://api.silentshield.io/api/v1/agent/telemetry"
const siteKey = "YOUR_API_KEY" // publishable — or os.Getenv("SILENTSHIELD_SITE_KEY")

var bots = []string{"GPTBot", "OAI-SearchBot", "ChatGPT-User", "ClaudeBot",
    "Claude-User", "PerplexityBot", "Perplexity-User", "Bingbot", "Applebot",
    "Amazonbot", "CCBot", "Bytespider", "meta-externalagent", "Google-Agent"}

func observe(next http.Handler) http.Handler {
    return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
        ua, signed := r.UserAgent(), r.Header.Get("Signature") != ""
        hit := signed
        for _, b := range bots {
            if strings.Contains(strings.ToLower(ua), strings.ToLower(b)) {
                hit = true
                break
            }
        }
        if hit {
            ip, _, _ := net.SplitHostPort(r.RemoteAddr)
            s := map[string]any{"ua": ua, "ip": ip, "path": r.URL.Path, "method": r.Method}
            if signed { // Web Bot Auth (RFC 9421) — forward for verification
                s["signature"] = r.Header.Get("Signature")
                s["signature_input"] = r.Header.Get("Signature-Input")
                s["signature_agent"] = r.Header.Get("Signature-Agent")
                s["authority"] = r.Host
            }
            body, _ := json.Marshal(map[string]any{"sightings": []any{s}})
            go func() { // best-effort, fail-open
                req, _ := http.NewRequest("POST", observeURL, bytes.NewReader(body))
                req.Header.Set("Content-Type", "application/json")
                req.Header.Set("x-api-key", siteKey)
                if resp, err := (&http.Client{Timeout: 3 * time.Second}).Do(req); err == nil {
                    resp.Body.Close()
                }
            }()
        }
        next.ServeHTTP(w, r)
    })
}

// net/http:  http.ListenAndServe(":8080", observe(mux))
// Gin:       srv := &http.Server{Addr: ":8080", Handler: observe(r)}
//            srv.ListenAndServe()

Preizkus povezave

Ni vam treba čakati na prvega pravega pajka UI, da preverite, ali vaša integracija deluje. Pošljite prazen paket — za vaš ključ registrira srčni utrip (heartbeat), ne da bi zapisal lažna opažanja:

Terminalbash
curl -X POST https://api.silentshield.io/api/v1/agent/telemetry \
  -H "Content-Type: application/json" \
  -H "x-api-key: YOUR_API_KEY" \
  --data '{"sightings":[]}'

Središče za agente vašega API ključa (API ključi → izberite ključ → Agent) se nato preklopi iz »ni povezano« v »povezano — posluša«. Vaše poročilo ostane prazno, dokler ne pride pravi agent: SilentShield si podatkov nikoli ne izmišljuje.

GDPR

Opazovanje temelji na zakonitem interesu (čl. 6(1)(f) GDPR). IP naslovi se na strežniški strani zgostijo s soljo, ki se menja vsak dan — surovi IP-ji se nikoli ne shranjujejo. Opažanja se hranijo 14 dni, shranjujejo pa se sploh samo zahteve znanih agentov UI: običajen promet obiskovalcev se nikoli ne beleži.

Pogosta vprašanja

Ali to upočasni moje spletno mesto?

Ne. Vsaka integracija je fire-and-forget: Next.js uporablja event.waitUntil(), PHP poroča šele po izročenem odgovoru, Node in Go pošiljata v ozadju. In če API SilentShield kdaj ne bi bil dosegljiv, vaše spletno mesto ostane neprizadeto — vse deluje fail-open.

Kaj šteje kot agent UI?

Samo zahteve, ki se ujemajo s SilentShieldovim imenikom botov znanih pajkov in agentov UI (GPTBot, ClaudeBot, PerplexityBot, …) ali ki nosijo podpis Web Bot Auth. Vedno aktualen seznam je na voljo na https://api.silentshield.io/api/v1/agent/bot-directory.

Zakaj še ne vidim podatkov?

Pajki UI prihajajo po svojem urniku. Na večini spletnih mest se prva opažanja pojavijo v 24–72 urah; pri manjših spletnih mestih lahko traja dlje. Medtem srčni utrip v vašem središču za agente potrjuje, da integracija sama deluje.

Naprej: uveljavljanje pravil za agente (2. stopnja) →