Data Pipeline Gap Analysis Last batch loaded: 2026-03-14 16:19 UTC
Total raw records
11,142
distinct records across all raw tables
What is counted

Unique records in the raw_data staging tables, before dbt. Each entity type uses a different deduplication key.

  • Posts2,967 distinct records
  • Users728 distinct records
  • Comments89 distinct records
  • Keywords7,072 distinct records
  • WhatsApp286 distinct records
Posts → COUNT(DISTINCT platform + COALESCE(content_id, link))
Users → COUNT(DISTINCT platform + user_id)
Comments → COUNT(DISTINCT comment_id)
Keywords / WhatsApp → COUNT(*) WHERE link ILIKE '%tiktok%'
Total loaded (final)
0
rows in social_data_alfa after dbt
What is counted

Row counts from the dbt-managed final tables in social_data_alfa. These are the records that passed all dbt transformations.

    Posts → social_data_alfa.posts
    Users → social_data_alfa.usernames
    Comments → social_data_alfa.comments
    Keywords → source_log WHERE source_type = 'keyword_script'
    WhatsApp → source_log WHERE source_type = 'whatsapp_script'
    Gap rows detected
    0
    raw records not found in final tables
    What is counted

    For each entity: raw LEFT JOIN final on business key, counting rows where the final side is NULL. Results are capped at 500 per entity — the true gap count may be higher.

    • Posts0 gap rows (capped at 500)
    • Users0 gap rows (capped at 500)
    • Comments0 gap rows (capped at 500)
    • Keywords0 gap rows (capped at 500)
    • WhatsApp0 gap rows (capped at 500)
    SELECT raw.* FROM raw
    LEFT JOIN final ON key
    WHERE final.id IS NULL LIMIT 500
    Avg load rate
    0.0%
    mean of per-entity load rates
    How this is calculated

    Each entity: loaded ÷ raw × 100. Then the mean of all 0 entity rates. Note: every entity is weighted equally, so Keywords at 0.5% pulls the average down as much as Posts at 100%.

      () ÷ 0 = 0.0%
      Avg. per-entity load rate
      0.0%
      Entity breakdown
      Posts tiktok_posts_meta → posts
      Raw 2,967
      Loaded —
      Gaps detected 0
      — loaded
      dbt not run

      No gaps detected.

      Users tiktok_users_meta → usernames
      Raw 728
      Loaded —
      Gaps detected 0
      — loaded
      dbt not run

      No gaps detected.

      Comments tiktok_comments_meta → comments
      Raw 89
      Loaded —
      Gaps detected 0
      — loaded
      dbt not run

      No gaps detected.

      Keywords tiktok_script_out → source_log
      Raw 7,072
      Loaded —
      Gaps detected 0
      — loaded
      dbt not run

      No gaps detected.

      WhatsApp whatsapp_script_out → source_log
      Raw 286
      Loaded —
      Gaps detected 0
      — loaded
      dbt not run

      No gaps detected.