music_library/docs/scrobble_rule_optimization.md

Scrobble Rule Application Optimization

Overview

This document describes the optimization made to the scrobble rule application system to improve performance when applying multiple rules.

Problem

Previously, when applying all enabled scrobble rules, each rule was applied independently with a separate database UPDATE query. With N rules, this resulted in N separate database operations, which became inefficient as the number of rules grew.

Previous Implementation

def apply_all_rules do
  list_enabled_rules()
  |> Enum.map(fn rule ->
    case apply_rule(rule) do
      {:ok, count} -> {:ok, {rule.type, rule.match_value, count}}
      {:error, reason} -> {:error, {rule.type, rule.match_value, reason}}
    end
  end)
end

This approach meant:

• 10 rules = 10 separate UPDATE queries
• 100 rules = 100 separate UPDATE queries
• Each query had to scan the entire scrobbled_tracks table

Solution

The optimization groups rules by type (album/artist) and applies all rules of each type in a single database query using SQLite's CASE statement.

New Implementation

def apply_all_rules do
  enabled_rules = list_enabled_rules()
  
  # Group rules by type
  {album_rules, artist_rules} = 
    Enum.split_with(enabled_rules, fn rule -> rule.type == :album end)
  
  # Apply all album rules in one query
  apply_all_album_rules(album_rules)
  
  # Apply all artist rules in one query
  apply_all_artist_rules(artist_rules)
end

This approach means:

• 10 album rules + 10 artist rules = 2 total UPDATE queries (one for albums, one for artists)
• 100 album rules + 100 artist rules = 2 total UPDATE queries
• Each query still scans the table once, but updates all matching records in a single pass

Technical Details

SQL Generation

The optimization dynamically builds a CASE statement for all rules of the same type:

Example for 2 album rules:

UPDATE scrobbled_tracks 
SET album = CASE 
  WHEN json_extract(album, '$.title') = 'Dark Side of the Moon' 
    THEN json_set(album, '$.musicbrainz_id', '12345678-1234-1234-1234-123456789012')
  WHEN json_extract(album, '$.title') = 'Wish You Were Here' 
    THEN json_set(album, '$.musicbrainz_id', 'abcdef12-3456-7890-abcd-ef1234567890')
  ELSE album 
END
WHERE json_extract(album, '$.title') IN ('Dark Side of the Moon', 'Wish You Were Here')

The WHERE clause uses IN to filter only tracks that match any of the rules, avoiding unnecessary updates.

Functions Added

1. apply_all_album_rules/1

Applies all album rules in a single query.

@spec apply_all_album_rules([ScrobbleRule.t()]) :: {:ok, non_neg_integer()} | {:error, any()}

2. apply_all_artist_rules/1

Applies all artist rules in a single query.

@spec apply_all_artist_rules([ScrobbleRule.t()]) :: {:ok, non_neg_integer()} | {:error, any()}

3. apply_all_album_rules/2

Applies all album rules to a specific set of tracks in a single query.

@spec apply_all_album_rules([ScrobbleRule.t()], [Track.t()]) :: 
  {:ok, non_neg_integer()} | {:error, any()}

4. apply_all_artist_rules/2

Applies all artist rules to a specific set of tracks in a single query.

@spec apply_all_artist_rules([ScrobbleRule.t()], [Track.t()]) :: 
  {:ok, non_neg_integer()} | {:error, any()}

Use Cases

The optimization handles two main use cases:

1. Periodic Full Application

The ApplyScrobbleRules Oban worker runs every 30 minutes and applies all rules to all tracks:

ScrobbleRules.apply_all_rules()

2. Incremental Application on New Tracks

When new tracks are inserted via LastFm.Feed.update/1, rules are applied to just those tracks:

tracks
|> ScrobbleRules.apply_all_rules()

Both use cases now benefit from batched rule application.

Performance Impact

Expected Improvements

For a database with:

• 10,000 scrobbled tracks
• 20 enabled rules (10 album + 10 artist)

Before:

• 20 UPDATE queries
• Each query scans ~10,000 rows
• Total: ~200,000 row scans

After:

• 2 UPDATE queries (1 for albums, 1 for artists)
• Each query scans ~10,000 rows
• Total: ~20,000 row scans

Result: ~10x reduction in database operations for this scenario.

The benefit scales with the number of rules:

• 5 rules: ~2.5x improvement
• 10 rules: ~5x improvement
• 20 rules: ~10x improvement
• 100 rules: ~50x improvement

Trade-offs

Return Value Semantics

With the batched approach, all rules of the same type report the same aggregate count (total rows updated) rather than individual per-rule counts:

# Before: Each rule gets its own count
[
  {:ok, {:album, "Album 1", 5}},   # 5 tracks updated
  {:ok, {:album, "Album 2", 3}},   # 3 tracks updated
]

# After: Both rules report the total updated
[
  {:ok, {:album, "Album 1", 8}},   # 8 total tracks updated
  {:ok, {:album, "Album 2", 8}},   # 8 total tracks updated
]

This is an acceptable trade-off because:

1. The logging still shows how many rules were applied
2. The total tracks updated is more meaningful for understanding impact
3. Getting individual counts would require separate queries, negating the optimization

Backward Compatibility

The API remains fully backward compatible:

• All existing functions (apply_rule/1, apply_album_rule/1, etc.) still work
• Return value format is unchanged: {:ok, {type, match_value, count}}
• Error handling is unchanged
• Tests continue to pass

Testing

New tests were added to verify the optimization:

1. apply_all_album_rules/1 applies multiple album rules in one query
2. apply_all_artist_rules/1 applies multiple artist rules in one query
3. Empty list handling returns {:ok, 0}
4. apply_all_rules/0 correctly batches rules by type
5. All existing tests continue to pass

Future Considerations

Potential Enhancements

1. Parallel Execution: Album and artist rule applications could run in parallel since they're independent
2. Query Optimization: Could add indexes on json_extract(album, '$.title') and json_extract(artist, '$.name') if performance is still a concern
3. Metrics: Add telemetry events to track batch sizes and execution times
4. Rule Ordering: If rule priority matters in the future, the CASE statement naturally handles this (first match wins)

Scalability

The current approach scales well up to ~100 rules per type. Beyond that, consider:

• SQL query length limits (SQLite's default limit is 1MB, which allows ~10,000 rules)
• Transaction size and memory usage
• Potential chunking if needed (apply rules in batches of N)

Conclusion

This optimization significantly improves the performance of scrobble rule application by reducing database operations from O(N) to O(1) per rule type, where N is the number of rules. The implementation maintains backward compatibility while providing substantial performance benefits that scale with the number of rules.