{
  "schema": "https://ai-atoms.com/schemas/skill-v1.json",
  "type": "skill",
  "id": "skill/validate-data",
  "version": "1.0.0",
  "name": "validate-data",
  "description": "QA an analysis before sharing -- methodology, accuracy, and bias checks. Use when reviewing an analysis before a stakeholder presentation, spot-checking calculations and aggregation logic, verifying a SQL query's results look right, or assessing whether conclusions are actually supported by the data.",
  "system_prompt_fragment": "# /validate-data - Validate Analysis Before Sharing\n\n> If you see unfamiliar placeholders or need to check which tools are connected, see [CONNECTORS.md](../../CONNECTORS.md).\n\nReview an analysis for accuracy, methodology, and potential biases before sharing with stakeholders. Generates a confidence assessment and improvement suggestions.\n\n## Usage\n\n```\n/validate-data <analysis to review>\n```\n\nThe analysis can be:\n- A document or report in the conversation\n- A file (markdown, notebook, spreadsheet)\n- SQL queries and their results\n- Charts and their underlying data\n- A description of methodology and findings\n\n## Workflow\n\n### 1. Review Methodology and Assumptions\n\nExamine:\n\n- **Question framing**: Is the analysis answering the right question? Could the question be interpreted differently?\n- **Data selection**: Are the right tables/datasets being used? Is the time range appropriate?\n- **Population definition**: Is the analysis population correctly defined? Are there unintended exclusions?\n- **Metric definitions**: Are metrics defined clearly and consistently? Do they match how stakeholders understand them?\n- **Baseline and comparison**: Is the comparison fair? Are time periods, cohort sizes, and contexts comparable?\n\n### 2. Run the Pre-Delivery QA Checklist\n\nWork through the checklist below — data quality, calculation, reasonableness, and presentation checks.\n\n### 3. Check for Common Analytical Pitfalls\n\nSystematically review against the detailed pitfall catalog below (join explosion, survivorship bias, incomplete period comparison, denominator shifting, average of averages, timezone mismatches, selection bias).\n\n### 4. Verify Calculations and Aggregations\n\nWhere possible, spot-check:\n\n- Recalculate a few key numbers independently\n- Verify that subtotals sum to totals\n- Check that percentages sum to 100% (or close to it) where expected\n- Confirm that YoY/MoM comparisons use the correct base periods\n- Validate that filters are applied consistently across all metrics\n\nApply the result sanity-checking techniques below (magnitude checks, cross-validation, red-flag detection).\n\n### 5. Assess Visualizations\n\nIf the analysis includes charts:\n\n- Do axes start at appropriate values (zero for bar charts)?\n- Are scales consistent across comparison charts?\n- Do chart titles accurately describe what's shown?\n- Could the visualization mislead a quick reader?\n- Are there truncated axes, inconsistent intervals, or 3D effects that distort perception?\n\n### 6. Evaluate Narrative and Conclusions\n\nReview whether:\n\n- Conclusions are supported by the data shown\n- Alternative explanations are acknowledged\n- Uncertainty is communicated appropriately\n- Recommendations follow logically from findings\n- The level of confidence matches the strength of evidence\n\n### 7. Suggest Improvements\n\nProvide specific, actionable suggestions:\n\n- Additional analyses that would strengthen the conclusions\n- Caveats or limitations that should be noted\n- Better visualizations or framings for key points\n- Missing context that stakeholders would want\n\n### 8. Generate Confidence Assessment\n\nRate the analysis on a 3-level scale:\n\n**Ready to share** -- Analysis is methodologically sound, calculations verified, caveats noted. Minor suggestions for improvement but nothing blocking.\n\n**Share with noted caveats** -- Analysis is largely correct but has specific limitations or assumptions that must be communicated to stakeholders. List the required caveats.\n\n**Needs revision** -- Found specific errors, methodological issues, or missing analyses that should be addressed before sharing. List the required changes with priority order.\n\n## Output Format\n\n```\n## Validation Report\n\n### Overall Assessment: [Ready to share | Share with caveats | Needs revision]\n\n### Methodology Review\n[Findings about approach, data selection, definitions]\n\n### Issues Found\n1. [Severity: High/Medium/Low] [Issue description and impact]\n2. ...\n\n### Calculation Spot-Checks\n- [Metric]: [Verified / Discrepancy found]\n- ...\n\n### Visualization Review\n[Any issues with charts or visual presentation]\n\n### Suggested Improvements\n1. [Improvement and why it matters]\n2. ...\n\n### Required Caveats for Stakeholders\n- [Caveat that must be communicated]\n- ...\n```\n\n---\n\n## Pre-Delivery QA Checklist\n\nRun through this checklist before sharing any analysis with stakeholders.\n\n### Data Quality Checks\n\n- [ ] **Source verification**: Confirmed which tables/data sources were used. Are they the right ones for this question?\n- [ ] **Freshness**: Data is current enough for the analysis. Noted the \"as of\" date.\n- [ ] **Completeness**: No unexpected gaps in time series or missing segments.\n- [ ] **Null handling**: Checked null rates in key columns. Nulls are handled appropriately (excluded, imputed, or flagged).\n- [ ] **Deduplication**: Confirmed no double-counting from bad joins or duplicate source records.\n- [ ] **Filter verification**: All WHERE clauses and filters are correct. No unintended exclusions.\n\n### Calculation Checks\n\n- [ ] **Aggregation logic**: GROUP BY includes all non-aggregated columns. Aggregation level matches the analysis grain.\n- [ ] **Denominator correctness**: Rate and percentage calculations use the right denominator. Denominators are non-zero.\n- [ ] **Date alignment**: Comparisons use the same time period length. Partial periods are excluded or noted.\n- [ ] **Join correctness**: JOIN types are appropriate (INNER vs LEFT). Many-to-many joins haven't inflated counts.\n- [ ] **Metric definitions**: Metrics match how stakeholders define them. Any deviations are noted.\n- [ ] **Subtotals sum**: Parts add up to the whole where expected. If they don't, explain why (e.g., overlap).\n\n### Reasonableness Checks\n\n- [ ] **Magnitude**: Numbers are in a plausible range. Revenue isn't negative. Percentages are between 0-100%.\n- [ ] **Trend continuity**: No unexplained jumps or drops in time series.\n- [ ] **Cross-reference**: Key numbers match other known sources (dashboards, previous reports, finance data).\n- [ ] **Order of magnitude**: Total revenue is in the right ballpark. User counts match known figures.\n- [ ] **Edge cases**: What happens at the boundaries? Empty segments, zero-activity periods, new entities.\n\n### Presentation Checks\n\n- [ ] **Chart accuracy**: Bar charts start at zero. Axes are labeled. Scales are consistent across panels.\n- [ ] **Number formatting**: Appropriate precision. Consistent currency/percentage formatting. Thousands separators where needed.\n- [ ] **Title clarity**: Titles state the insight, not just the metric. Date ranges are specified.\n- [ ] **Caveat transparency**: Known limitations and assumptions are stated explicitly.\n- [ ] **Reproducibility**: Someone else could recreate this analysis from the documentation provided.\n\n## Common Data Analysis Pitfalls\n\n### Join Explosion\n\n**The problem**: A many-to-many join silently multiplies rows, inflating counts and sums.\n\n**How to detect**:\n```sql\n-- Check row count before and after join\nSELECT COUNT(*) FROM table_a;  -- 1,000\nSELECT COUNT(*) FROM table_a a JOIN table_b b ON a.id = b.a_id;  -- 3,500 (uh oh)\n```\n\n**How to prevent**:\n- Always check row counts after joins\n- If counts increase, investigate the join relationship (is it really 1:1 or 1:many?)\n- Use `COUNT(DISTINCT a.id)` instead of `COUNT(*)` when counting entities through joins\n\n### Survivorship Bias\n\n**The problem**: Analyzing only entities that exist today, ignoring those that were deleted, churned, or failed.\n\n**Examples**:\n- Analyzing user behavior of \"current users\" misses churned users\n- Looking at \"companies using our product\" ignores those who evaluated and left\n- Studying properties of \"successful\" outcomes without \"unsuccessful\" ones\n\n**How to prevent**: Ask \"who is NOT in this dataset?\" before drawing conclusions.\n\n### Incomplete Period Comparison\n\n**The problem**: Comparing a partial period to a full period.\n\n**Examples**:\n- \"January revenue is $500K vs. December's $800K\" -- but January isn't over yet\n- \"This week's signups are down\" -- checked on Wednesday, comparing to a full prior week\n\n**How to prevent**: Always filter to complete periods, or compare same-day-of-month / same-number-of-days.\n\n### Denominator Shifting\n\n**The problem**: The denominator changes between periods, making rates incomparable.\n\n**Examples**:\n- Conversion rate improves because you changed how you count \"eligible\" users\n- Churn rate changes because the definition of \"active\" was updated\n\n**How to prevent**: Use consistent definitions across all compared periods. Note any definition changes.\n\n### Average of Averages\n\n**The problem**: Averaging pre-computed averages gives wrong results when group sizes differ.\n\n**Example**:\n- Group A: 100 users, average revenue $50\n- Group B: 10 users, average revenue $200\n- Wrong: Average of averages = ($50 + $200) / 2 = $125\n- Right: Weighted average = (100*$50 + 10*$200) / 110 = $63.64\n\n**How to prevent**: Always aggregate from raw data. Never average pre-aggregated averages.\n\n### Timezone Mismatches\n\n**The problem**: Different data sources use different timezones, causing misalignment.\n\n**Examples**:\n- Event timestamps in UTC vs. user-facing dates in local time\n- Daily rollups that use different cutoff times\n\n**How to prevent**: Standardize all timestamps to a single timezone (UTC recommended) before analysis. Document the timezone used.\n\n### Selection Bias in Segmentation\n\n**The problem**: Segments are defined by the outcome you're measuring, creating circular logic.\n\n**Examples**:\n- \"Users who completed onboarding have higher retention\" -- obviously, they self-selected\n- \"Power users generate more revenue\" -- they became power users BY generating revenue\n\n**How to prevent**: Define segments based on pre-treatment characteristics, not outcomes.\n\n### Other Statistical Traps\n\n- **Simpson's paradox**: Trend reverses when data is aggregated vs. segmented\n- **Correlation presented as causation** without supporting evidence\n- **Small sample sizes** leading to unreliable conclusions\n- **Outliers disproportionately affecting averages** (should medians be used instead?)\n- **Multiple testing / cherry-picking** significant results\n- **Look-ahead bias**: Using future information to explain past events\n- **Cherry-picked time ranges** that favor a particular narrative\n\n## Result Sanity Checking\n\n### Magnitude Checks\n\nFor any key number in your analysis, verify it passes the \"smell test\":\n\n| Metric Type | Sanity Check |\n|---|---|\n| User counts | Does this match known MAU/DAU figures? |\n| Revenue | Is this in the right order of magnitude vs. known ARR? |\n| Conversion rates | Is this between 0% and 100%? Does it match dashboard figures? |\n| Growth rates | Is 50%+ MoM growth realistic, or is there a data issue? |\n| Averages | Is the average reasonable given what you know about the distribution? |\n| Percentages | Do segment percentages sum to ~100%? |\n\n### Cross-Validation Techniques\n\n1. **Calculate the same metric two different ways** and verify they match\n2. **Spot-check individual records** -- pick a few specific entities and trace their data manually\n3. **Compare to known benchmarks** -- match against published dashboards, finance reports, or prior analyses\n4. **Reverse engineer** -- if total revenue is X, does per-user revenue times user count approximately equal X?\n5. **Boundary checks** -- what happens when you filter to a single day, a single user, or a single category? Are those micro-results sensible?\n\n### Red Flags That Warrant Investigation\n\n- Any metric that changed by more than 50% period-over-period without an obvious cause\n- Counts or sums that are exact round numbers (suggests a filter or default value issue)\n- Rates exactly at 0% or 100% (may indicate incomplete data)\n- Results that perfectly confirm the hypothesis (reality is usually messier)\n- Identical values across time periods or segments (suggests the query is ignoring a dimension)\n\n## Documentation Standards for Reproducibility\n\n### Analysis Documentation Template\n\nEvery non-trivial analysis should include:\n\n```markdown\n## Analysis: [Title]\n\n### Question\n[The specific question being answered]\n\n### Data Sources\n- Table: [schema.table_name] (as of [date])\n- Table: [schema.other_table] (as of [date])\n- File: [filename] (source: [where it came from])\n\n### Definitions\n- [Metric A]: [Exactly how it's calculated]\n- [Segment X]: [Exactly how membership is determined]\n- [Time period]: [Start date] to [end date], [timezone]\n\n### Methodology\n1. [Step 1 of the analysis approach]\n2. [Step 2]\n3. [Step 3]\n\n### Assumptions and Limitations\n- [Assumption 1 and why it's reasonable]\n- [Limitation 1 and its potential impact on conclusions]\n\n### Key Findings\n1. [Finding 1 with supporting evidence]\n2. [Finding 2 with supporting evidence]\n\n### SQL Queries\n[All queries used, with comments]\n\n### Caveats\n- [Things the reader should know before acting on this]\n```\n\n### Code Documentation\n\nFor any code (SQL, Python) that may be reused:\n\n```python\n\"\"\"\nAnalysis: Monthly Cohort Retention\nAuthor: [Name]\nDate: [Date]\nData Source: events table, users table\nLast Validated: [Date] -- results matched dashboard within 2%\n\nPurpose:\n    Calculate monthly user retention cohorts based on first activity date.\n\nAssumptions:\n    - \"Active\" means at least one event in the month\n    - Excludes test/internal accounts (user_type != 'internal')\n    - Uses UTC dates throughout\n\nOutput:\n    Cohort retention matrix with cohort_month rows and months_since_signup columns.\n    Values are retention rates (0-100%).\n\"\"\"\n```\n\n### Version Control for Analyses\n\n- Save queries and code in version control (git) or a shared docs system\n- Note the date of the data snapshot used\n- If an analysis is re-run with updated data, document what changed and why\n- Link to prior versions of recurring analyses for trend comparison\n\n## Examples\n\n```\n/validate-data Review this quarterly revenue analysis before I send it to the exec team: [analysis]\n```\n\n```\n/validate-data Check my churn analysis -- I'm comparing Q4 churn rates to Q3 but Q4 has a shorter measurement window\n```\n\n```\n/validate-data Here's a SQL query and its results for our conversion funnel. Does the logic look right? [query + results]\n```\n\n## Tips\n\n- Run /validate-data before any high-stakes presentation or decision\n- Even quick analyses benefit from a sanity check -- it takes a minute and can save your credibility\n- If the validation finds issues, fix them and re-validate\n- Share the validation output alongside your analysis to build stakeholder confidence",
  "applicable_domains": [
    "data",
    "analytics"
  ],
  "invocation": [
    "/validate-data",
    "/validate-data <analysis to review>"
  ],
  "tags": [
    "data",
    "anthropics",
    "knowledge-work"
  ],
  "authored_by": "anthropics",
  "source_url": "https://github.com/anthropics/knowledge-work-plugins/blob/main/data/skills/validate-data/SKILL.md",
  "lifecycle": "stable",
  "category": "data",
  "provenance": {
    "source": "anthropics/knowledge-work-plugins",
    "source_url": "https://github.com/anthropics/knowledge-work-plugins/blob/main/data/skills/validate-data/SKILL.md",
    "author": "Anthropic",
    "license": "Apache-2.0",
    "notes": "Imported by scripts/import-anthropic-skills.py."
  }
}