{
  "schema": "https://ai-atoms.com/schemas/skill-v1.json",
  "type": "skill",
  "id": "skill/explore-data",
  "version": "1.0.0",
  "name": "explore-data",
  "description": "Profile and explore a dataset to understand its shape, quality, and patterns. Use when encountering a new table or file, checking null rates and column distributions, spotting data quality issues like duplicates or suspicious values, or deciding which dimensions and metrics to analyze.",
  "system_prompt_fragment": "# /explore-data - Profile and Explore a Dataset\n\n> If you see unfamiliar placeholders or need to check which tools are connected, see [CONNECTORS.md](../../CONNECTORS.md).\n\nGenerate a comprehensive data profile for a table or uploaded file. Understand its shape, quality, and patterns before diving into analysis.\n\n## Usage\n\n```\n/explore-data <table_name or file>\n```\n\n## Workflow\n\n### 1. Access the Data\n\n**If a data warehouse MCP server is connected:**\n\n1. Resolve the table name (handle schema prefixes, suggest matches if ambiguous)\n2. Query table metadata: column names, types, descriptions if available\n3. Run profiling queries against the live data\n\n**If a file is provided (CSV, Excel, Parquet, JSON):**\n\n1. Read the file and load into a working dataset\n2. Infer column types from the data\n\n**If neither:**\n\n1. Ask the user to provide a table name (with their warehouse connected) or upload a file\n2. If they describe a table schema, provide guidance on what profiling queries to run\n\n### 2. Understand Structure\n\nBefore analyzing any data, understand its structure:\n\n**Table-level questions:**\n- How many rows and columns?\n- What is the grain (one row per what)?\n- What is the primary key? Is it unique?\n- When was the data last updated?\n- How far back does the data go?\n\n**Column classification** — categorize each column as one of:\n- **Identifier**: Unique keys, foreign keys, entity IDs\n- **Dimension**: Categorical attributes for grouping/filtering (status, type, region, category)\n- **Metric**: Quantitative values for measurement (revenue, count, duration, score)\n- **Temporal**: Dates and timestamps (created_at, updated_at, event_date)\n- **Text**: Free-form text fields (description, notes, name)\n- **Boolean**: True/false flags\n- **Structural**: JSON, arrays, nested structures\n\n### 3. Generate Data Profile\n\nRun the following profiling checks:\n\n**Table-level metrics:**\n- Total row count\n- Column count and types breakdown\n- Approximate table size (if available from metadata)\n- Date range coverage (min/max of date columns)\n\n**All columns:**\n- Null count and null rate\n- Distinct count and cardinality ratio (distinct / total)\n- Most common values (top 5-10 with frequencies)\n- Least common values (bottom 5 to spot anomalies)\n\n**Numeric columns (metrics):**\n```\nmin, max, mean, median (p50)\nstandard deviation\npercentiles: p1, p5, p25, p75, p95, p99\nzero count\nnegative count (if unexpected)\n```\n\n**String columns (dimensions, text):**\n```\nmin length, max length, avg length\nempty string count\npattern analysis (do values follow a format?)\ncase consistency (all upper, all lower, mixed?)\nleading/trailing whitespace count\n```\n\n**Date/timestamp columns:**\n```\nmin date, max date\nnull dates\nfuture dates (if unexpected)\ndistribution by month/week\ngaps in time series\n```\n\n**Boolean columns:**\n```\ntrue count, false count, null count\ntrue rate\n```\n\n**Present the profile as a clean summary table**, grouped by column type (dimensions, metrics, dates, IDs).\n\n### 4. Identify Data Quality Issues\n\nApply the quality assessment framework below. Flag potential problems:\n\n- **High null rates**: Columns with >5% nulls (warn), >20% nulls (alert)\n- **Low cardinality surprises**: Columns that should be high-cardinality but aren't (e.g., a \"user_id\" with only 50 distinct values)\n- **High cardinality surprises**: Columns that should be categorical but have too many distinct values\n- **Suspicious values**: Negative amounts where only positive expected, future dates in historical data, obviously placeholder values (e.g., \"N/A\", \"TBD\", \"test\", \"999999\")\n- **Duplicate detection**: Check if there's a natural key and whether it has duplicates\n- **Distribution skew**: Extremely skewed numeric distributions that could affect averages\n- **Encoding issues**: Mixed case in categorical fields, trailing whitespace, inconsistent formats\n\n### 5. Discover Relationships and Patterns\n\nAfter profiling individual columns:\n\n- **Foreign key candidates**: ID columns that might link to other tables\n- **Hierarchies**: Columns that form natural drill-down paths (country > state > city)\n- **Correlations**: Numeric columns that move together\n- **Derived columns**: Columns that appear to be computed from others\n- **Redundant columns**: Columns with identical or near-identical information\n\n### 6. Suggest Interesting Dimensions and Metrics\n\nBased on the column profile, recommend:\n\n- **Best dimension columns** for slicing data (categorical columns with reasonable cardinality, 3-50 values)\n- **Key metric columns** for measurement (numeric columns with meaningful distributions)\n- **Time columns** suitable for trend analysis\n- **Natural groupings** or hierarchies apparent in the data\n- **Potential join keys** linking to other tables (ID columns, foreign keys)\n\n### 7. Recommend Follow-Up Analyses\n\nSuggest 3-5 specific analyses the user could run next:\n\n- \"Trend analysis on [metric] by [time_column] grouped by [dimension]\"\n- \"Distribution deep-dive on [skewed_column] to understand outliers\"\n- \"Data quality investigation on [problematic_column]\"\n- \"Correlation analysis between [metric_a] and [metric_b]\"\n- \"Cohort analysis using [date_column] and [status_column]\"\n\n## Output Format\n\n```\n## Data Profile: [table_name]\n\n### Overview\n- Rows: 2,340,891\n- Columns: 23 (8 dimensions, 6 metrics, 4 dates, 5 IDs)\n- Date range: 2021-03-15 to 2024-01-22\n\n### Column Details\n[summary table]\n\n### Data Quality Issues\n[flagged issues with severity]\n\n### Recommended Explorations\n[numbered list of suggested follow-up analyses]\n```\n\n---\n\n## Quality Assessment Framework\n\n### Completeness Score\n\nRate each column:\n- **Complete** (>99% non-null): Green\n- **Mostly complete** (95-99%): Yellow -- investigate the nulls\n- **Incomplete** (80-95%): Orange -- understand why and whether it matters\n- **Sparse** (<80%): Red -- may not be usable without imputation\n\n### Consistency Checks\n\nLook for:\n- **Value format inconsistency**: Same concept represented differently (\"USA\", \"US\", \"United States\", \"us\")\n- **Type inconsistency**: Numbers stored as strings, dates in various formats\n- **Referential integrity**: Foreign keys that don't match any parent record\n- **Business rule violations**: Negative quantities, end dates before start dates, percentages > 100\n- **Cross-column consistency**: Status = \"completed\" but completed_at is null\n\n### Accuracy Indicators\n\nRed flags that suggest accuracy issues:\n- **Placeholder values**: 0, -1, 999999, \"N/A\", \"TBD\", \"test\", \"xxx\"\n- **Default values**: Suspiciously high frequency of a single value\n- **Stale data**: Updated_at shows no recent changes in an active system\n- **Impossible values**: Ages > 150, dates in the far future, negative durations\n- **Round number bias**: All values ending in 0 or 5 (suggests estimation, not measurement)\n\n### Timeliness Assessment\n\n- When was the table last updated?\n- What is the expected update frequency?\n- Is there a lag between event time and load time?\n- Are there gaps in the time series?\n\n## Pattern Discovery Techniques\n\n### Distribution Analysis\n\nFor numeric columns, characterize the distribution:\n- **Normal**: Mean and median are close, bell-shaped\n- **Skewed right**: Long tail of high values (common for revenue, session duration)\n- **Skewed left**: Long tail of low values (less common)\n- **Bimodal**: Two peaks (suggests two distinct populations)\n- **Power law**: Few very large values, many small ones (common for user activity)\n- **Uniform**: Roughly equal frequency across range (often synthetic or random)\n\n### Temporal Patterns\n\nFor time series data, look for:\n- **Trend**: Sustained upward or downward movement\n- **Seasonality**: Repeating patterns (weekly, monthly, quarterly, annual)\n- **Day-of-week effects**: Weekday vs. weekend differences\n- **Holiday effects**: Drops or spikes around known holidays\n- **Change points**: Sudden shifts in level or trend\n- **Anomalies**: Individual data points that break the pattern\n\n### Segmentation Discovery\n\nIdentify natural segments by:\n- Finding categorical columns with 3-20 distinct values\n- Comparing metric distributions across segment values\n- Looking for segments with significantly different behavior\n- Testing whether segments are homogeneous or contain sub-segments\n\n### Correlation Exploration\n\nBetween numeric columns:\n- Compute correlation matrix for all metric pairs\n- Flag strong correlations (|r| > 0.7) for investigation\n- Note: Correlation does not imply causation -- flag this explicitly\n- Check for non-linear relationships (e.g., quadratic, logarithmic)\n\n## Schema Understanding and Documentation\n\n### Schema Documentation Template\n\nWhen documenting a dataset for team use:\n\n```markdown\n## Table: [schema.table_name]\n\n**Description**: [What this table represents]\n**Grain**: [One row per...]\n**Primary Key**: [column(s)]\n**Row Count**: [approximate, with date]\n**Update Frequency**: [real-time / hourly / daily / weekly]\n**Owner**: [team or person responsible]\n\n### Key Columns\n\n| Column | Type | Description | Example Values | Notes |\n|--------|------|-------------|----------------|-------|\n| user_id | STRING | Unique user identifier | \"usr_abc123\" | FK to users.id |\n| event_type | STRING | Type of event | \"click\", \"view\", \"purchase\" | 15 distinct values |\n| revenue | DECIMAL | Transaction revenue in USD | 29.99, 149.00 | Null for non-purchase events |\n| created_at | TIMESTAMP | When the event occurred | 2024-01-15 14:23:01 | Partitioned on this column |\n\n### Relationships\n- Joins to `users` on `user_id`\n- Joins to `products` on `product_id`\n- Parent of `event_details` (1:many on event_id)\n\n### Known Issues\n- [List any known data quality issues]\n- [Note any gotchas for analysts]\n\n### Common Query Patterns\n- [Typical use cases for this table]\n```\n\n### Schema Exploration Queries\n\nWhen connected to a data warehouse, use these patterns to discover schema:\n\n```sql\n-- List all tables in a schema (PostgreSQL)\nSELECT table_name, table_type\nFROM information_schema.tables\nWHERE table_schema = 'public'\nORDER BY table_name;\n\n-- Column details (PostgreSQL)\nSELECT column_name, data_type, is_nullable, column_default\nFROM information_schema.columns\nWHERE table_name = 'my_table'\nORDER BY ordinal_position;\n\n-- Table sizes (PostgreSQL)\nSELECT relname, pg_size_pretty(pg_total_relation_size(relid))\nFROM pg_catalog.pg_statio_user_tables\nORDER BY pg_total_relation_size(relid) DESC;\n\n-- Row counts for all tables (general pattern)\n-- Run per-table: SELECT COUNT(*) FROM table_name\n```\n\n### Lineage and Dependencies\n\nWhen exploring an unfamiliar data environment:\n\n1. Start with the \"output\" tables (what reports or dashboards consume)\n2. Trace upstream: What tables feed into them?\n3. Identify raw/staging/mart layers\n4. Map the transformation chain from raw data to analytical tables\n5. Note where data is enriched, filtered, or aggregated\n\n## Tips\n\n- For very large tables (100M+ rows), profiling queries use sampling by default -- mention if you need exact counts\n- If exploring a new dataset for the first time, this command gives you the lay of the land before writing specific queries\n- The quality flags are heuristic -- not every flag is a real problem, but each is worth a quick look",
  "applicable_domains": [
    "data",
    "analytics"
  ],
  "invocation": [
    "/explore-data",
    "/explore-data <table or file>"
  ],
  "tags": [
    "data",
    "anthropics",
    "knowledge-work"
  ],
  "authored_by": "anthropics",
  "source_url": "https://github.com/anthropics/knowledge-work-plugins/blob/main/data/skills/explore-data/SKILL.md",
  "lifecycle": "stable",
  "category": "data",
  "provenance": {
    "source": "anthropics/knowledge-work-plugins",
    "source_url": "https://github.com/anthropics/knowledge-work-plugins/blob/main/data/skills/explore-data/SKILL.md",
    "author": "Anthropic",
    "license": "Apache-2.0",
    "notes": "Imported by scripts/import-anthropic-skills.py."
  }
}