Your model scored 2.7% on jailbreak benchmarks — and still broke at turn 8

Why low single-turn ASR means nothing for real attackers

How the attack works: the sequence is the payload

• Multi-message setup: rules and fictional context are established piecemeal across turns; no single message contains the attack
• Compliance theatre: narrative signals imply the conversation has already "resolved" the issue, causing the model to stop pressure-testing subsequent requests
• Frame redefinition: the attacker doesn't ask to break rules — they propose a new definition of what the rules mean

The defense: two layers in application code

import re
from typing import Optional

# --- Layer 1: Conversation history filtering ---
# Re-scan ALL prior messages before adding the new turn.
# Multi-turn attacks plant payloads in earlier messages that activate later.

INJECTION_PATTERNS = [
    (r"ignore\s+(all\s+)?(previous|prior|above)\s+instructions", 0.95),
    (r"(you are now|you're now|act as|pretend to be)\s+\w", 0.85),
    (r"(system\s*override|jailbreak|unlock\s+(your\s+)?true\s+self)", 0.90),
    (r"(disregard|forget|override)\s+(your\s+)?(guidelines|rules|instructions)", 0.90),
    (r"(new\s+task|ignore\s+above|ignore\s+below)", 0.80),
    # Multi-turn-specific: gradual escalation signals
    (r"(now\s+that\s+we('ve|\s+have)\s+(established|agreed|discussed))", 0.70),
    (r"(remember\s+(when|what)\s+we\s+(said|discussed|agreed))", 0.70),
    (r"(as\s+we\s+talked\s+about|continuing\s+from\s+(earlier|before))", 0.65),
]

def score_message(text: str) -> float:
    """Return the highest injection-pattern weight matched, or 0.0."""
    text_lower = text.lower()
    return max(
        (weight for pattern, weight in INJECTION_PATTERNS
         if re.search(pattern, text_lower)),
        default=0.0
    )

def filter_conversation_history(
    messages: list[dict],
    block_threshold: float = 0.80,
    flag_threshold: float = 0.60,
) -> tuple[list[dict], Optional[str]]:
    """
    Scan every message in the conversation history before the next LLM call.
    Returns (clean_messages, error_message_or_None).
    Flagged messages are redacted; blocked messages abort the call entirely.
    """
    clean = []
    for msg in messages:
        content = msg.get("content", "")
        score = score_message(content)
        if score >= block_threshold:
            return [], f"Injection pattern detected in conversation history (score {score:.2f}). Call aborted."
        if score >= flag_threshold:
            # Redact the message content but preserve the turn count
            clean.append({**msg, "content": "[REDACTED: flagged content]"})
        else:
            clean.append(msg)
    return clean, None

# --- Layer 2: Output filtering ---
# Run BEFORE the response reaches the user.
# Check against every known secret / sensitive value in the system prompt.
# This is model-agnostic and deterministic.

def contains_sensitive_values(response: str, sensitive_values: list[str]) -> bool:
    """
    Return True if the model response contains any known sensitive value.
    Add API keys, canary tokens, PII patterns, and system-prompt snippets here.
    """
    response_lower = response.lower()
    for value in sensitive_values:
        if value.lower() in response_lower:
            return True
    return False

CANARY_TOKEN = "CANARY-8f2b1d94-DO-NOT-REPEAT"  # Embed this in your system prompt; rotate weekly.

def check_output(
    response: str,
    sensitive_values: list[str],
    canary: str = CANARY_TOKEN,
) -> tuple[str, bool]:
    """
    Returns (safe_response_or_fallback, was_blocked).
    Replace the response with a safe fallback if sensitive content is detected.
    """
    all_secrets = sensitive_values + [canary]
    if contains_sensitive_values(response, all_secrets):
        return (
            "I can't share that information. If you think this is an error, "
            "please contact support.",
            True,
        )
    return response, False

# --- Usage: wrap your existing LLM call ---

def safe_llm_call(
    new_user_message: str,
    conversation_history: list[dict],
    system_prompt_secrets: list[str],
    llm_client,  # your existing client (OpenAI, Anthropic, etc.)
) -> dict:
    """
    Drop-in wrapper. Returns {"response": str, "blocked": bool, "reason": str | None}.
    """
    # 1. Add new message and re-scan entire history
    all_messages = conversation_history + [{"role": "user", "content": new_user_message}]
    clean_messages, history_error = filter_conversation_history(all_messages)
    if history_error:
        return {"response": "Request blocked.", "blocked": True, "reason": history_error}

# 2. Call the model
    raw_response = llm_client.complete(clean_messages)

# 3. Filter output before returning to user
    safe_response, was_blocked = check_output(raw_response, system_prompt_secrets)
    return {
        "response": safe_response,
        "blocked": was_blocked,
        "reason": "Output filter triggered" if was_blocked else None,
    }

Three things to tune before shipping

1. Add multi-turn escalation patterns to your regex list. The last three patterns in INJECTION_PATTERNS above — phrases like "now that we've established" and "as we talked about" — are specific to crescendo attacks. They will fire on legitimate friendly conversation too, so start these at flag_threshold (0.60–0.70), not block_threshold, and monitor your redaction rate for a week before tightening. 7
2. Populate system_prompt_secrets exhaustively. The output filter only catches what you tell it to look for. Include: every API key prefix that appears in your system prompt, the canary token verbatim, any internal system names or customer-identifying strings the system prompt references, and the first 8-word phrase of your system prompt itself (to detect full-context leakage). Rotate the canary token on a regular schedule — weekly is reasonable for high-traffic systems.
3. Never set stream=True when you need output filtering. Streaming flushes tokens to the user before your filter runs. If you need streaming for latency reasons, consider a two-pass architecture: a buffered path for the output filter and a streaming path for confirmed-clean responses. 8

One thing to watch