ConvPreserve
GitHub

Testing Conversation Preservation Across LLM APIs

API gateways, proxies, and SDK adapters let applications switch between OpenAI, Anthropic, and Gemini—but a translated request or response can pass schema validation (HTTP 200) while silently dropping prompts, stripping tool error flags, masking stop reasons, or injecting parameters. ConvPreserve compares payloads before and after translation to catch silent semantic changes offline, with no API keys, no LLM judge, and no reference outputs required.

Get Started View on GitHub Paper
Supported APIs: OpenAI Chat Completions · OpenAI Responses · Anthropic Messages · Google Gemini
Covers: Requests · Responses · Streaming (SSE)
Source RequestAnthropic Messages

          

          

            
Translated RequestGoogle Gemini

            

          

        

        

          

            Schema validation
            Valid (HTTP 200)
            Accepted by the Gemini API without error
          

          

            ConvPreserve
            Silent Loss (T1)
            System prompt "Reply in JSON." dropped during translation
          

        

      

    

  







  

    
Getting Started

    

      

        

          1. Install & compare two payloads
          
        

        
$ pip install git+https://github.com/convpreserve/conv-preserve.git
$ convpreserve check --side request \
    --src-dialect openai --dst-dialect anthropic \
    before.json after.json
S08  T1  unlicensed control thinking injected  dst:/thinking

        
Python 3.11+ · zero dependencies · exits with 0 when preserved or 1 on silent loss.

      


      

        

          2. Or test a running local gateway
          
        

        
# Point your gateway's upstream URL at http://127.0.0.1:18600
$ convpreserve probe --gateway http://127.0.0.1:8080 --target openai

        
Starts a local mock upstream server and sends 51 built-in test requests—no API keys needed.

      

    

  







  

    
Documentation

    

      

        
Finding Categories

        

          

            
              
TierRulesClassificationWhat It ChecksDefault Exit

            
            
              

                T1
                S01–S13
                Silent semantic loss
                Avoidable changes to prompts, message roles, tool schemas, tool calls or error flags, sampling/reasoning parameters, or stop reasons.
                exit 1
              

              

                T2
                D01–D06
                Target API limitation
                Target API lacks a direct equivalent (e.g., system merged into user message), translator fills in a default parameter, or cache/usage metadata changes.
                exit 0
              

              

                T3
                R01–R08
                Equivalent format difference
                Different JSON structure with identical conversation meaning, such as empty string vs. null or merged adjacent text blocks.
                exit 0
              

            
          

        

      


      

        
CLI Reference

        

          

            
              
CommandArgumentsPurpose

            
            
              

                convpreserve check
                --side {request,response,stream} --src-dialect FMT --dst-dialect FMT BEFORE AFTER
                Compare a payload before and after translation (formats: openai | responses | anthropic | gemini; pass --json for JSON output).
              

              

                convpreserve batch
                MANIFEST.jsonl [--out FILE] [--summary-md FILE]
                Check a batch of payload pairs listed in a .jsonl file for regression suites and CI pipelines.
              

              

                convpreserve probe
                --gateway URL --target {openai,responses,gemini,anthropic}
                Test a running local gateway end-to-end: starts a local mock upstream server (127.0.0.1:18600), sends 51 built-in test requests, and checks every forwarded payload.
              

              

                convpreserve explain
                RULE_ID | --list
                Show what a rule checks and known bug patterns in open-source translators (for example, convpreserve explain S08).
              

            
          

        

      

    

  







  

    
Examples

    

      

        
        
        
        
      

      

        

          
Source RequestAnthropic Messages

          

        

        

          
Translated RequestOpenAI Chat

          

        

      

      

        

          S03 · T1
          
        

        
      

    

  







  

    
Evaluation

    

      

        
76.0%

        
recall · 2.0% false-alarm rate

        
Catches 95 of 125 injected translation bugs on held-out tests, with only 2 false alarms across 100 valid format conversions.

      

      

        
18.5%

        
127 / 685 translated requests

        
Requests across 5 popular translators that pass schema validation yet silently alter prompts, roles, or tools.

      

      

        
36.4%

        
203 / 558 translated streams

        
Translated SSE streams that crash or silently corrupt messages when parsed by an official OpenAI SDK.

      

      

        
7 fixed

        
26 bugs reported · 12 projects

        
Confirmed translation bugs reported to open-source gateway and SDK maintainers, with 7 fixes already merged.

      

    

  







  

    
Paper

    

      

        

          
Testing Conversation Preservation Across LLM APIs: A Rule-Based Oracle for Detecting Silent Information Loss

          
Anonymous authors · Under review (2026)

        

        

          Code, test suites, and bundled examples: github.com/convpreserve/conv-preserve