Carmee Agentic Assistant - Shadow Mode Testing

Project: Build AI agent to draft sales replies for Carmee in Zoho Cliq
Phase: Phase 0 - Intelligence Engine Validation (Shadow Mode)
Timeline: Week of Feb 17-22, 2026
Owner: Minnie (Main session)
Stakeholder: Quan Gan

---

Objective

Validate agentic intelligence engine can outperform Klansys's custom GPT for pathway classification and draft generation before exposing to Carmee.

Approach: Parallel shadow testing - generate drafts invisibly, compare quality, iterate until validated.

---

Current State

Carmee's workflow:

• Works in Zoho Cliq (not Telegram)
• Uses custom GPT on ChatGPT (Klansys-built) for draft generation
• Uses programmatic routing workflow (rigid rules, brittle on edge cases)
• Manual context switching: Cliq → ChatGPT → Zoho CRM

Problems to solve:

1. Custom GPT can't pull live Zoho CRM data (manual copy/paste)
2. Programmatic routing breaks on edge cases (no natural language understanding)
3. No learning loop (doesn't adapt to Carmee's style over time)

---

Phase 0: Shadow Mode Validation

Goals

1. Prove agentic engine generates better drafts than custom GPT
2. Validate pathway classification accuracy (targets 1-4: 80% of inquiries)
3. Quantify time saved per inquiry
4. Build confidence before Carmee integration

Architecture (No Cliq Integration Yet)

Inputs:

• Sample inquiries (5-10 real examples Carmee handled)
• Zoho CRM deal data (via API)
• Google Drive meeting notes (already have access)
• Gmail thread context (already have read-only access)

Processing:

1. Agentic layer analyzes inquiry
2. Classifies pathway (1-7)
3. Determines stakeholder type, decision cycle
4. Pulls relevant context (CRM, Drive, Gmail)
5. Generates draft + guidance notes

Outputs:

• Draft reply (markdown formatted)
• Pathway classification + confidence score
• Guidance notes: "Pathway 3 (City/Municipal), 45-day cycle, emphasize grant alignment"
• Suggested next steps: "Follow up in 7 days, escalate to Kristin if budget concerns"

Comparison:

• Log my output vs. Klansys's GPT output
• Quan manually reviews quality
• Track accuracy metrics per inquiry

---

Integration Requirements

1. Zoho CRM API OAuth (IN PROGRESS)

Status: Setup guide to be created (similar to Gmail OAuth)

Required scopes:

• ZohoCRM.modules.deals.READ - Pull deal data
• ZohoCRM.modules.contacts.READ - Customer history
• ZohoCRM.modules.notes.READ - Deal notes/context

Endpoints:

• /crm/v3/Deals - Deal pipeline data
• /crm/v3/Contacts - Contact records
• /crm/v3/Notes - Internal notes on deals

Credentials location: /home/node/.openclaw/credentials/zoho-crm-tokens.json

Setup steps: (To be documented in setup guide)

1. Create Zoho API client in Zoho Developer Console
2. Configure OAuth redirect URL
3. Request required scopes
4. Generate authorization URL
5. Exchange auth code for tokens
6. Store refresh token for automated renewal

2. Existing Integrations (Already Working)

• ✅ Google Drive API: Meeting notes, proposals
• ✅ Gmail API: Email thread context (3 accounts authorized)

---

Testing Protocol

Sample Inquiry Structure

For each test inquiry, log:

## Test #N - [Brief Description]

**Date:** YYYY-MM-DD
**Pathway (Actual):** [1-7]
**Customer Type:** [School District / Nonprofit / City / Camp / etc.]

**Raw Inquiry:**
[Paste full inquiry text]

**CRM Context Pulled:**
- Deal ID: [if available]
- Contact: [name, role]
- Pipeline stage: [Qualification / Proposal / etc.]
- Past interactions: [summary]

**My Output:**

**Pathway Classification:** [1-7] (Confidence: X%)
**Draft Reply:**
[Full draft text]

**Guidance Notes:**
[Pathway description, decision cycle, emphasis points]

**Suggested Next Steps:**
[Follow-up timing, escalation triggers]

**Klansys GPT Output:**
[Paste custom GPT output for comparison]

**Carmee's Final Sent Version:**
[Paste what Carmee actually sent]

**Quan's Assessment:**
- Pathway accuracy: ✅ / ❌
- Draft quality: [1-5 rating]
- Context relevance: [1-5 rating]
- Time saved estimate: [minutes]
- Notes: [Any feedback]

---

Success Criteria

Minimum thresholds to proceed to Cliq integration:

• Pathway classification accuracy: ≥90% (9/10 correct)
• Draft quality rating: ≥4.0 avg (Quan's subjective assessment)
• Context relevance: ≥4.0 avg (pulls right data from CRM/Drive/Gmail)
• Time saved: ≥10 min/inquiry vs. current workflow

Target: 10 shadow tests across pathways 1-4 before going live.

---

Pathways Reference (For Classification)

Pathway 1: Grant-Funded School

• K-12 public schools with ELOP, 21st CCLC, ASAP, or similar grants
• Decision cycle: 30-90 days (grant-dependent)
• Emphasis: Grant alignment, educational outcomes, compliance

Pathway 2: 3rd-Party Nonprofit

• Nonprofits serving schools (Boys & Girls Clubs, YMCAs, etc.)
• Decision cycle: 30-60 days
• Emphasis: Program impact, community reach, partnership model

Pathway 3: City/Municipal

• Parks & Rec departments, city after-school programs
• Decision cycle: 45-90 days (municipal procurement)
• Emphasis: Public funding justification, community benefit, scalability

Pathway 4: Camps

• Summer camps, day camps, specialty camps
• Decision cycle: 15-45 days (seasonal urgency)
• Emphasis: Engagement, safety, quick setup, seasonal ROI

Pathways 5-7: (Lower volume, defer to Phase 2)

• 5: Operator Partnership (FECs, mobile operators)
• 6: Pilot Program (schools testing before commitment)
• 7: Professional/Operators (distributors, training orgs)

---

Next Steps

Immediate (This Week: Feb 17-22)

1. Create Zoho CRM OAuth setup guide (Minnie)
2. Authorize Zoho API access (Quan)
3. Pull 5-10 sample inquiries (Quan forwards, or Minnie pulls from CRM)
4. Generate shadow drafts (Minnie)
5. Log comparison results (Minnie)
6. Quan reviews quality (Quan assessment)

Week 2 (Feb 25-28) - If Validated

1. Set up Zoho Cliq bot integration
2. Deploy shadow mode in Cliq (parallel with custom GPT)
3. Carmee tests both outputs (blind A/B)
4. Measure: draft quality, time saved, edit rate

Week 3+ (Mar 2+) - If Successful

1. Deprecate custom GPT
2. Full Cliq integration live
3. Learning loop active (tracks edits, adapts style)
4. Expand to pathways 5-7

---

Risk Mitigation

Risk: Agent output is worse than custom GPT

• Mitigation: Shadow mode catches this before Carmee sees it
• Fallback: Keep custom GPT, iterate on agent, test again

Risk: CRM API rate limits or access issues

• Mitigation: Cache deal data, handle API errors gracefully
• Fallback: Manual inquiry forwarding (no CRM pull) for testing

Risk: Carmee resists new tool

• Mitigation: Prove quality first in shadow mode, then parallel A/B
• Adoption: Carmee chooses which output to use (not forced)

Risk: Klansys feels sidelined

• Mitigation: Frame as "building on her foundation," minimal involvement until validated
• Communication: Update her once agent is proven, acknowledge her custom GPT work

---

Logging

All test results logged in this file below this line.

---

Test Results

Test #1 - [Pending]

(To be populated after Zoho OAuth setup + first sample inquiry)

---

Document Metadata

• Created: February 17, 2026, 2:07 AM UTC
• Owner: Minnie (Main session)
• Status: Setup phase (awaiting Zoho OAuth authorization)
• Next review: After first 3 shadow tests completed