Git

Latest log messages

• 9e7fc4be931407f2720f33a8596e86ac2c3d49ba on 02/01/2026 3:33 pm by Zohaib Sibte Hassan <[email protected]> [GitHub Diff]
  Message: Feature: Add correctness verification test suite and utf8mb4 support
  
  Correctness Test Suite (examples/correctness-test/):
  - collect-bluesky-posts.ts: Downloads posts from Bluesky Jetstream WebSocket
  - correctness-test.ts: Verifies data consistency across 3 primaries + 1 replica
  - Phase 1: Initial load of 5K+ posts
  - Phase 2: Random deletes with verification
  - Phase 3: Random updates with verification
  - Data integrity check comparing local SQLite vs cluster
  - Optional high-load stress test mode
  - run-correctness-test.sh: Wrapper script for easy execution
  
  MySQL Protocol utf8mb4 Fix (protocol/server.go):
  - Changed charset from 33 (utf8_general_ci) to 45 (utf8mb4_general_ci)
  - Enables full 4-byte UTF-8 support including emojis and all Unicode
  - Fixed in handshake packet and column definitions
  
  ???? Generated with [Claude Code](https://claude.com/claude-code)
  
  Co-Authored-By: Claude Opus 4.5 <[email protected]>
• d2843748f32f9555e5d6d1f6d71309aba8ffba91 on 02/01/2026 2:55 pm by Zohaib Sibte Hassan <[email protected]> [GitHub Diff]
  Message: Refactor: Update cluster-with-replicas script for dynamic config generation
  
  - Generate node and replica configs dynamically with auto-generated PSK
  - Both replicas now use follow_addresses for transparent failover
  - Consistent style with run-cluster.sh and start-replica.sh
  
  ???? Generated with [Claude Code](https://claude.com/claude-code)
  
  Co-Authored-By: Claude Opus 4.5 <[email protected]>
• 0d08e60be575693693c58c3ad1f371d124f8b06c on 02/01/2026 2:48 pm by Zohaib Sibte Hassan <[email protected]> [GitHub Diff]
  Message: Refactor: Improve example scripts with auto-generated configs and PSK
  
  - run-cluster.sh now generates configs dynamically with replica secret
  - start-replica.sh builds, cleans up, and auto-generates PSK
  - Both scripts share consistent PSK for seamless replica connection
  - Added --replica-secret flag to run-cluster.sh
  - Cleaned up output formatting to match run-cluster.sh style
  
  ???? Generated with [Claude Code](https://claude.com/claude-code)
  
  Co-Authored-By: Claude Opus 4.5 <[email protected]>
• 52cacfd399e49a5550004eaf7f2665894bce33a3 on 02/01/2026 2:43 pm by Zohaib Sibte Hassan <[email protected]> [GitHub Diff]
  Message: Docs: Update README with new replica failover configuration
  
  Update replica mode documentation to reflect the new multi-node
  discovery and transparent failover features:
  - Replace master_address with follow_addresses array examples
  - Add discovery_interval_seconds and failover_timeout_seconds
  - Document CLI flag --follow-addresses
  - Describe automatic failover behavior
  
  ???? Generated with [Claude Code](https://claude.com/claude-code)
  
  Co-Authored-By: Claude Opus 4.5 <[email protected]>
• 7681feb32302d7286699da65a2e56ce5d36227e9 on 02/01/2026 2:41 pm by Zohaib Sibte Hassan <[email protected]> [GitHub Diff]
  Message: Feature: Add cluster discovery and transparent failover for replicas
  
  Replicas can now discover cluster nodes and automatically failover
  when their current source becomes unavailable.
  
  Changes:
  - Add GetClusterNodes RPC for cluster membership discovery
  - Replace single master_address with follow_addresses array
  - Add --follow-addresses CLI flag for comma-separated seed nodes
  - Implement discovery loop polling nodes periodically (max 3 per interval)
  - Implement transparent failover with configurable timeout
  - Use incarnation-based merge for cluster view consistency
  - Add discovery_interval_seconds and failover_timeout_seconds config
  - Update examples and documentation
  
  ???? Generated with [Claude Code](https://claude.com/claude-code)
  
  Co-Authored-By: Claude Opus 4.5 <[email protected]>
• 7d60d1af1f7ebb489c9f4ee583fc823d7ece42ee on 02/01/2026 1:25 pm by Zohaib Sibte Hassan <[email protected]> [GitHub Diff]
  Message: Fix: TxnID-ordered streaming for replica failover correctness
  
  The previous SeqNum-based transaction index caused ordering inconsistencies
  across nodes because SeqNum is node-local. When transactions from different
  coordinators (with clock skew) commit in interleaved order, the SeqNum
  iteration order doesn't match the TxnID filter criteria, causing replicas
  to miss transactions on reconnect/failover.
  
  Changes:
  - Add /txn_idx/txnid/ index ordered by TxnID (globally unique, HLC-based)
  - ScanTransactions now iterates by TxnID order instead of SeqNum
  - CommitTransaction and StoreReplayedTransaction write to both indexes
  - SeqNum index retained for backward compatibility and GC
  
  This enables safe replica failover: any cluster node can serve
  StreamChanges(fromTxnID=X) with consistent ordering.
  
  ???? Generated with [Claude Code](https://claude.com/claude-code)
  
  Co-Authored-By: Claude Opus 4.5 <[email protected]>
• d47b4554edb1b43e331257f051dae9c3910e4f24 on 02/01/2026 5:58 am by Zohaib Sibte Hassan <[email protected]> [GitHub Diff]
  Message: Updating README.md with new features
• 61d6832ba7adc22d3dd3b1c1aa4352d295f1a814 on 01/01/2026 11:07 am by Zohaib Sibte Hassan <[email protected]> [GitHub Diff]
  Message: Perf: Reduce network payload in 2PC COMMIT phase
  
  Optimize the distributed commit mechanism by removing redundant transaction
  statements and CDC data from the Phase 2 (COMMIT) request. Since this data is
  already persisted as "Write Intents" during Phase 1 (PREPARE), it is not needed
  on the wire during commit.
  
  - Coordinator: Stop sending Statements in PhaseCommit requests.
  - Engine: Update CommitResult to return DDL SQL extracted from persisted intents.
  - Handler: Use engine-provided DDL SQL for schema version increments instead of
  inspecting the network payload.
  - Tests: Update write coordinator tests to reflect the optimized payload.
  
  This significantly reduces bandwidth usage for multi-row transactions.
• 4eaa438ff3f8d378c66293e8263304790fdadb35 on 28/12/2025 7:51 am by Zohaib Sibte Hassan <[email protected]> [GitHub Diff]
  Message: Feature: Add Unix socket support for MySQL server
  
  Enable MySQL server to listen on both TCP and Unix socket simultaneously.
  When unix_socket config is empty, only TCP is used (existing behavior).
  
  - Add UnixSocket and UnixSocketPerm fields to MySQLConfiguration
  - Modify MySQLServer to support multiple listeners
  - Cleanup stale socket on startup, remove socket on shutdown
  - Default socket permissions: 0660
  
  ???? Generated with [Claude Code](https://claude.com/claude-code)
  
  Co-Authored-By: Claude Opus 4.5 <[email protected]>
• 0a676fd28de20d7a5a20a28362e32ee075ab0eea on 19/12/2025 7:01 pm by Zohaib Sibte Hassan <[email protected]> [GitHub Diff]
  Message: Refactor: Remove CDC pipeline buffering for streaming
  
  Remove ProcessCDCEntries and related merge/grouping infrastructure
  that was blocking streaming. CDC entries are now converted directly
  to statements inline without intermediate buffering.
  
  - Remove GroupByIntentKey, MergeGroup, ValidateCDCEntry
  - Remove CDCPipelineConfig, CDCPipelineResult types
  - Keep only ConvertToStatement for O(1) per-entry conversion
  - Update handler.go to convert entries inline
  - Delete 1300+ lines of pipeline code and tests
  
  Memory: 134MB -> ~0 for 100k row transactions
  
  ???? Generated with [Claude Code](https://claude.com/claude-code)
  
  Co-Authored-By: Claude Opus 4.5 <[email protected]>
• 17cd00dceff5197a9610e37fccd5a0d9ca5d69dd on 19/12/2025 8:36 am by Zohaib Sibte Hassan <[email protected]> [GitHub Diff]
  Message: Refactor: Binary intent key encoding for performance
  
  Convert intent keys from string format to binary encoding:
  - Version byte + table prefix + typed PK values
  - DDL keys: 0xFE marker + table name
  - DB-op keys: 0xFF marker + database name
  - Use PK declaration order (not alphabetical)
  - Precompute IntentKeyPrefix in SchemaCache
  - Base64url encoding at JSON/HTTP boundaries
  - XXH64 hashing for conflict detection
  
  ???? Generated with [Claude Code](https://claude.com/claude-code)
  
  Co-Authored-By: Claude Opus 4.5 <[email protected]>
• 0c9281a080aa72a8c082f17d64bfe13521fb0cdf on 18/12/2025 5:57 pm by Zohaib Sibte Hassan <[email protected]> [GitHub Diff]
  Message:
• 52570dd419db84dcf37040150fe140ac7d09a350 on 18/12/2025 5:33 pm by Zohaib Sibte Hassan <[email protected]> [GitHub Diff]
  Message: Duh
• 19bf0cf3f05abda851e912b654723309eafe5c36 on 18/12/2025 5:29 pm by Zohaib Sibte Hassan <[email protected]> [GitHub Diff]
  Message: Improvements improvements
• 1fbc885ef912c43c00e2afd1efa5b8b41400fb82 on 18/12/2025 5:10 pm by Zohaib Sibte Hassan <[email protected]> [GitHub Diff]
  Message: Revert: Use mimalloc for native memory allocation
  
  Confirmed mimalloc was not causing memory leaks - memory was just
  invisible to Go pprof because it's CGO memory. Reverting debug
  changes that used Go heap allocation.
  
  ???? Generated with [Claude Code](https://claude.com/claude-code)
  
  Co-Authored-By: Claude Opus 4.5 <[email protected]>
• 0197f9e00bf28f039a8c2e9dd381f0cfd394c8b8 on 18/12/2025 3:23 pm by Zohaib Sibte Hassan <[email protected]> [GitHub Diff]
  Message: mimalloc packaging
• ec2c4dcb035b67e662ff02c33b15056dd3325a67 on 18/12/2025 3:03 pm by Zohaib Sibte Hassan <[email protected]> [GitHub Diff]
  Message: Refactor: Replace external go-mimalloc with internal pkg/mimalloc
  
  - Switch encoding/native.go to use internal github.com/maxpert/marmot/pkg/mimalloc
  - Remove github.com/mxmauro/go-mimalloc dependency
  - Promote xsync/v3 to direct dependency
  
  ???? Generated with [Claude Code](https://claude.com/claude-code)
  
  Co-Authored-By: Claude Opus 4.5 <[email protected]>
• 35f23bca29fe593d629b82386c448aee840fcc18 on 18/12/2025 11:23 am by Zohaib Sibte Hassan <[email protected]> [GitHub Diff]
  Message: Refactor: Unified TableSchema eliminates duplicate PRAGMA parsers
  
  - Create db/schema.go with canonical TableSchema and ColumnSchema types
  - Extend SchemaCache to extract full column metadata (type, nullable, PK)
  - Migrate DatabaseManager.GetTableSchema to use cached schema
  - Migrate coordinator to use cached schema for auto-increment detection
  - Migrate replica/stream_client.go to use unified CDC applier
  - Remove protocol/schema_provider.go (deprecated duplicate PRAGMA parser)
  - Remove protocol/intent_key_compat_test.go (tested deleted function)
  
  Net reduction: 656 lines (908 deleted, 252 added)
  
  ???? Generated with [Claude Code](https://claude.com/claude-code)
  
  Co-Authored-By: Claude Opus 4.5 <[email protected]>
• 79d18a21eb53b8e72957dd37b9bde0288bcc50b8 on 18/12/2025 10:33 am by Zohaib Sibte Hassan <[email protected]> [GitHub Diff]
  Message: Refactor: Unified CDC applier consolidating 4 duplicate implementations
  
  - Create db/cdc_applier.go with CDCExecutor and CDCSchemaProvider interfaces
  - Fix TEXT type affinity bug: 3 of 4 CDC paths stored TEXT as BLOB
  - Add comprehensive test suite (559 lines) with type affinity verification
  - Remove duplicate CDC apply logic from:
  - db/transaction.go (-171 lines)
  - db/sqlite_batch_committer.go (-131 lines)
  - grpc/replication_handler.go (-169 lines)
  - grpc/delta_sync.go (-175 lines)
  - Net production code reduction: ~474 lines
  
  ???? Generated with [Claude Code](https://claude.com/claude-code)
  
  Co-Authored-By: Claude Opus 4.5 <[email protected]>
• caacb1a199d5c593233e424e49d379ba09199eac on 18/12/2025 9:46 am by Zohaib Sibte Hassan <[email protected]> [GitHub Diff]
  Message: Fix: Critical safety fixes and dead code removal
  
  Phase 0 - Safety Fixes:
  - Add defer to mutex Lock/Unlock in protocol/server.go (deadlock prevention)
  - Add defer to RUnlock in coordinator/handler.go
  - Fix file handle leak in grpc/server.go snapshot streaming
  - Fix goroutine leak in publisher/worker.go Start/Stop race condition
  - Add 30s context timeout in db/sqlite_batch_committer.go
  
  Phase 1 - Dead Code Removal:
  - Remove unused CapturedRow struct from db/preupdate_hook.go
  - Remove IntentStore interface and IntentMeta struct from db/memory_stores.go
  - Remove XsyncIntentStore implementation from db/memory_stores_xsync.go
  - Remove IntentStore tests from db/memory_stores_xsync_test.go
  
  Net reduction: ~300 lines removed
  
  ???? Generated with [Claude Code](https://claude.com/claude-code)
  
  Co-Authored-By: Claude Opus 4.5 <[email protected]>
• 34f038e2f5d49a8f5372cbb576d9ec7520de0f06 on 18/12/2025 8:50 am by Zohaib Sibte Hassan <[email protected]> [GitHub Diff]
  Message: Refactor: Consolidate CDCEntry type and cleanup duplicates
  
  - Create common/cdc_entry.go as single source of truth for CDCEntry
  - Remove duplicate GenericCDCEntry from coordinator and publisher
  - Simplify publisherAdapter by eliminating field-by-field conversion
  - Add captured_row.go for hook capture abstraction
  - Add memory stores with xsync-based lock-free implementations
  - Update all package references to use common.CDCEntry
  
  ???? Generated with [Claude Code](https://claude.com/claude-code)
  
  Co-Authored-By: Claude Opus 4.5 <[email protected]>
• e664fb5d71ef2b3e4b092e039c5d47e202f740c2 on 18/12/2025 4:00 am by Zohaib Sibte Hassan <[email protected]> [GitHub Diff]
  Message: Fix: Remove dead batch writer and fix CDC raw row cleanup
  
  - Remove unused batch writer system (batchCh, pebbleBatchOp, batchWriter goroutine)
  - Fix AbortTransaction to call DeleteCapturedRows for orphaned raw entries
  - Fix CleanupAfterCommit to also delete /cdc/raw/ prefix entries
  - Remove unused IsBusy method and isMetaStoreBusy helper
  
  ???? Generated with [Claude Code](https://claude.com/claude-code)
  
  Co-Authored-By: Claude Opus 4.5 <[email protected]>
• f497242c61e8265290b794b3026b1bf0fc3655b8 on 17/12/2025 6:04 am by Zohaib Sibte Hassan <[email protected]> [GitHub Diff]
  Message: Perf: Native memory allocation and hookDB connection optimization
  
  Two performance improvements:
  
  1. Native memory allocation via mimalloc for hot paths:
  - Add encoding/native.go with MarshalNative using mimalloc
  - Add encoding/native_stub.go for non-CGO fallback
  - Integrate in meta_store_pebble.go hot paths (WriteIntent, BeginTransaction, etc.)
  - 75% reduction in Go heap allocations for serialization
  
  2. Fix hookDB connection held too long during CDC processing:
  - Release hookDB connection immediately after SQLite rollback
  - ProcessCapturedRows now runs after connection release (uses only Pebble)
  - Reduces hookDB hold time from seconds (Pebble warmup) to milliseconds
  - Fixes initial stall with high concurrency (512 threads)
  
  ???? Generated with [Claude Code](https://claude.com/claude-code)
  
  Co-Authored-By: Claude Opus 4.5 <[email protected]>
• fded2a22df9de10547dd993f14ebf35332951f88 on 16/12/2025 7:27 pm by Zohaib Sibte Hassan <[email protected]> [GitHub Diff]
  Message: Perf: Eliminate double serialization and merge cleanup operations
  
  P3: WriteIntentEntry now accepts map[string][]byte directly
  - Removed marshal at caller (preupdate_hook, replication_engine)
  - Removed unnecessary unmarshal in WriteIntentEntry
  - Single Marshal call for entire IntentEntry struct
  - ~50% reduction in WriteIntentEntry allocations
  
  P4: New CleanupAfterCommit merges 3 operations into 2 commits
  - Phase 1: GC markers (must be visible before deletions)
  - Phase 2: Single batch for all deletions (intents + CDC)
  - Reduced from 3 commits to 2 commits
  - ~43% reduction in cleanupAfterCommit CPU
  
  ???? Generated with [Claude Code](https://claude.com/claude-code)
  
  Co-Authored-By: Claude Opus 4.5 <[email protected]>
• 655a208add8065096a3b61f1c6b64134616f4f87 on 16/12/2025 6:16 pm by Zohaib Sibte Hassan <[email protected]> [GitHub Diff]
  Message: Perf: Eliminate read-modify-write anti-patterns in PebbleMetaStore
  
  Profiling showed 23% CPU in cleanupAfterCommit due to toxic
  read-unmarshal-modify-marshal-write cycles. This commit implements
  three optimizations:
  
  P0: GC Intent Markers
  - MarkIntentsForCleanup writes presence keys (/gc/intent/) instead of
  modifying intent records
  - writeIntentSlowPath checks GC markers before expensive unmarshal
  
  P1: Transaction Record Split
  - Split TransactionRecord into immutable (/txn/) and commit (/txn_commit/)
  - Status stored as 1-byte key (/txn_status/)
  - BeginTransaction/CommitTransaction no longer do read-modify-write
  - GetTransaction reconstructs for backward compatibility
  
  P2: Raw Heartbeat Optimization
  - Heartbeat stored as raw 8-byte timestamp (/txn_heartbeat/)
  - Heartbeat() reduced from 16-line read-modify-write to single write
  - WriteIntent paths refresh heartbeat on success
  - CleanupStaleTransactions reads 8 bytes instead of full record
  
  Performance impact:
  - Heartbeat: ~1057 bytes → 8 bytes (132x smaller)
  - CommitTransaction: No read-unmarshal-modify-marshal cycle
  - Status updates: 1-byte writes instead of full record
  
  ???? Generated with [Claude Code](https://claude.com/claude-code)
  
  Co-Authored-By: Claude Opus 4.5 <[email protected]>
• 422da37ed4e31651cba5b88a9e3b2ab8445110e4 on 16/12/2025 2:34 pm by Zohaib Sibte Hassan <[email protected]> [GitHub Diff]
  Message: Fix: Schema version race condition causing Node 3 instability
  
  Prevent nodes from becoming ALIVE before their schema versions catch up
  with peers. This fixes a race condition where anti-entropy could sync
  from behind peers, causing DDL loss and node instability.
  
  Defense-in-depth approach:
  - Promotion criteria now blocks JOINING→ALIVE when local schema < peer
  - Anti-entropy skips peers with older schema versions
  - Fresh databases (schema=0) are now properly compared
  
  ???? Generated with [Claude Code](https://claude.com/claude-code)
  
  Co-Authored-By: Claude Opus 4.5 <[email protected]>
• 774b65fa8d23aba3a43345f99b13ad2d1257adf2 on 16/12/2025 1:18 pm by Zohaib Sibte Hassan <[email protected]> [GitHub Diff]
  Message: Refactor: Revert PebbleDB tuning and simplify preupdate hook
  
  - Revert PebbleDB tuning options (bloom filters, WAL sync intervals,
  table cache shards) that were causing cluster instability
  - Simplify preupdate hook implementation
  - Add schema cache for better column metadata handling
  - Consolidate test files for hook capture and row processing
  - Remove unused conflict detection test file
  - Clean up coordinator handler and transaction code
  
  ???? Generated with [Claude Code](https://claude.com/claude-code)
  
  Co-Authored-By: Claude Opus 4.5 <[email protected]>
• 59e05f9a3bca6983714eda584c114d8f658c28d4 on 16/12/2025 4:15 am by Zohaib Sibte Hassan <[email protected]> [GitHub Diff]
  Message: Refactor: Use centralized encoding package for msgpack
  
  Replace direct vmihailenco/msgpack imports with the centralized
  encoding package to ensure consistent serialization across codebase.
  
  ???? Generated with [Claude Code](https://claude.com/claude-code)
  
  Co-Authored-By: Claude Opus 4.5 <[email protected]>
• 97896463069cb28abcd788fc55d91cddd432be0e on 15/12/2025 7:57 pm by Zohaib Sibte Hassan <[email protected]> [GitHub Diff]
  Message:
• 9cba252c24b22466028139cfe4b66893ce01ec4a on 15/12/2025 7:33 pm by Zohaib Sibte Hassan <[email protected]> [GitHub Diff]
  Message: Refactor WriteCoordinator with comprehensive unit tests
  
  Extract reusable components from WriteCoordinator:
  - ClusterState: encapsulates cluster topology and quorum calculation
  - TxnMetrics: centralized transaction telemetry recording
  - Error types: PrepareConflictError, QuorumNotAchievedError,
  CoordinatorNotParticipatedError, PartialCommitError
  
  Add comprehensive unit test suite (142 tests + 5 benchmarks):
  - Builder methods (validateStatements, buildPrepareRequest, buildCommitRequest)
  - Prepare phase (runPreparePhase, executePreparePhase)
  - Commit phase (sendRemoteCommits, waitForRemoteQuorum, commitLocalAfterRemoteQuorum)
  - Integration tests for full transaction state machines
  - Network partition and split-brain prevention tests
  - Concurrency and race condition tests (10 dedicated tests)
  
  Fix code quality issues:
  - Race condition in TestWaitForCondition (use atomic.Int32)
  - Unchecked errors in abort path (add explicit ignore with comment)
  - Remove unused fields from test helpers
  - Fix impossible len() < 0 condition
  - Fix error comparison in classifyPrepareError (use type assertion)
  
  All tests pass with -race flag.
  
  ???? Generated with [Claude Code](https://claude.com/claude-code)
  
  Co-Authored-By: Claude Opus 4.5 <[email protected]>
• a97fe285a6f90fff5f7daa1f7ce207ef2e9e4ea2 on 15/12/2025 4:25 pm by Zohaib Sibte Hassan <[email protected]> [GitHub Diff]
  Message: Refactor: Extract ReplicationEngine to eliminate code duplication
  
  LocalReplicator and ReplicationHandler had ~640 lines of duplicated
  prepare/commit/abort logic. This refactoring:
  
  - Extracts common replication logic into ReplicationEngine (477 lines)
  - Reduces LocalReplicator from 414 → 84 lines (80% reduction)
  - Reduces ReplicationHandler from 1104 → 635 lines (42% reduction)
  - Adds comprehensive test coverage (775 lines, 13 tests)
  
  Fixes included:
  - DDL schema version tracking: Check for DDL before commit (transaction
  removed after commit, breaking version increments on followers)
  - GC race condition: Add closed checks in PebbleMetaStore cleanup methods
  - Refactor monolithic Prepare() into focused private methods
  - Remove broken error handling in WriteIntent (string-matching SQLite
  errors when MetaStore is PebbleDB - dead code that never executed)
  
  All tests passing. No behavioral changes to replication protocol.
  
  ???? Generated with [Claude Code](https://claude.com/claude-code)
  
  Co-Authored-By: Claude Sonnet 4.5 <[email protected]>
• 8b86d3cf9cd630926232424dcaad88fcaa9b0627 on 15/12/2025 9:21 am by Zohaib Sibte Hassan <[email protected]> [GitHub Diff]
  Message:
• 591f1fa8191282118dff2e80adcce6424fbfdf8c on 15/12/2025 9:15 am by Zohaib Sibte Hassan <[email protected]> [GitHub Diff]
  Message: Optimize single-node example config for benchmarks
  
  Align run-single-node.sh config with cluster node configs:
  - Replace [mvcc] with [transaction] section
  - Add query_pipeline, metastore, batch_commit sections
  - Increase coordinator timeouts to 120s
  - Enable batch committer with adaptive sizing
  - Switch to production logging (json, non-verbose)
  
  ???? Generated with [Claude Code](https://claude.com/claude-code)
  
  Co-Authored-By: Claude Opus 4.5 <[email protected]>
• 7ebbf4403ff7e2450c94be48cdd9de49f342b54e on 15/12/2025 8:59 am by Zohaib Sibte Hassan <[email protected]> [GitHub Diff]
  Message: Optimize default config for production performance
  
  pprof analysis revealed console logging consuming 21% of CPU time.
  Changes reduce logging overhead by 40x (21% → 0.5%).
  
  Changes:
  - Default logging format: "console" → "json" (21% CPU savings)
  - Default metastore cache: 64MB → 128MB (reduce Pebble disk reads)
  - Example configs: verbose=false, format="json", cache=128MB
  
  Benchmark Results:
  - Before: zerolog.ConsoleWriter.Write 16.52s (21.09%)
  - After: zerolog (all) 0.39s (0.53%)
  - Total CPU samples reduced 78.33s → 73.57s (6% reduction)
  
  ???? Generated with [Claude Code](https://claude.com/claude-code)
  
  Co-Authored-By: Claude Opus 4.5 <[email protected]>
• 0c8f7f6ceb75f22ae00ecae9a2354e08d1cf689b on 15/12/2025 8:49 am by Zohaib Sibte Hassan <[email protected]> [GitHub Diff]
  Message: Implement adaptive checkpoint strategy for SQLite batch committer
  
  Add intelligent WAL checkpoint management to reduce p99 flush latency from
  ~100ms to ≤10ms by moving checkpoints to background goroutines.
  
  Core Features:
  - WAL size monitoring via fast file stat (~10μs syscall)
  - Adaptive checkpoint modes: PASSIVE (4MB) and RESTART (16MB) thresholds
  - Background checkpoint execution (non-blocking)
  - Dynamic batch sizing (2x during checkpoint to exploit parallel window)
  - Timer-based flush disabled during checkpoint to prevent interference
  - 7 new Prometheus metrics for checkpoint observability
  
  Implementation:
  - Add checkpoint configuration to BatchCommitConfiguration
  - Extend SQLiteBatchCommitter with checkpoint state tracking (atomic.Bool)
  - Implement checkWALSize() for fast WAL file size checks
  - Implement backgroundCheckpoint() with PRAGMA wal_checkpoint and metrics
  - Integrate checkpoint trigger after tx.Commit() before promise resolution
  - Update NewSQLiteBatchCommitter signature with checkpoint parameters
  - Update all callers (db_integration.go, tests)
  
  Metrics Added:
  - batch_committer_checkpoint_total (counter by mode)
  - batch_committer_checkpoint_duration_ms (histogram)
  - batch_committer_wal_size_mb (histogram)
  - batch_committer_wal_frames_log (histogram)
  - batch_committer_wal_frames_checkpointed (histogram)
  - batch_committer_checkpoint_busy_total (counter)
  - batch_committer_checkpoint_efficiency (histogram)
  
  Benchmark Results (200K transactions):
  - 8,132 PASSIVE checkpoints (Node 1), zero RESTART checkpoints
  - WAL kept at 4-8MB (never hit 16MB emergency threshold)
  - Checkpoint p99 duration: ~10ms
  - Checkpoint efficiency: 99% (nearly all frames checkpointed)
  - Dynamic batching working: batches reaching 200 (2x max_batch_size)
  
  Test Coverage:
  - TestBatchCommitter_NoCheckpointBelowThreshold
  - TestBatchCommitter_PassiveCheckpointTriggered
  - TestBatchCommitter_RestartCheckpointTriggered
  - TestBatchCommitter_DynamicBatchSizing
  - TestBatchCommitter_TimerDisabledDuringCheckpoint
  - TestBatchCommitter_ConcurrentOperationsWithCheckpoint
  - TestBatchCommitter_CheckpointMetrics
  
  All tests pass including race detection.
  
  ???? Generated with [Claude Code](https://claude.com/claude-code)
  
  Co-Authored-By: Claude Sonnet 4.5 <[email protected]>
• f88cbe1d522feaf4fee3eb2dad7eab949738a8b1 on 15/12/2025 8:21 am by Zohaib Sibte Hassan <[email protected]> [GitHub Diff]
  Message: Add SQLite batch committer for improved write throughput
  
  - Implement batching of CDC commits with configurable batch size and wait time
  - Add immediate flush channel for size-based triggers
  - Add metrics: flush counter (by trigger), batch size, flush duration
  - Configure optimized SQLite PRAGMAs (synchronous=OFF, mmap, WAL)
  - Set wal_autocheckpoint to 1000 pages (~4MB) for faster checkpoints
  - Add batch_commit configuration section to example configs
  
  ???? Generated with [Claude Code](https://claude.com/claude-code)
  
  Co-Authored-By: Claude Sonnet 4.5 <[email protected]>
• c123f42ed2e1a86eadd5a2368746828cfe4319b9 on 14/12/2025 8:30 pm by Zohaib Sibte Hassan <[email protected]> [GitHub Diff]
  Message: Add SQLite batch committer for improved write throughput
  
  - New SQLiteBatchCommitter batches multiple DML transactions into single
  SQLite transaction, reducing fsync overhead
  - Uses go-future library for Promise/Future pattern
  - Opens dedicated connection with aggressive write optimizations:
  - synchronous=OFF, cache_size=64MB, mmap_size=256MB
  - wal_autocheckpoint=10000, journal_size_limit=64MB
  - DDL transactions bypass batching (require immediate schema reload)
  - Config: batch_commit.enabled=true, max_batch_size=512, max_wait_ms=2
  - Reuses existing SchemaCache.Reload() via conn.Raw() for DRY
  - Deletes unused commit_batcher.go (was dead code)
  - Fixes bug: batch committer properly stopped in Close()
  
  ???? Generated with [Claude Code](https://claude.com/claude-code)
  
  Co-Authored-By: Claude Opus 4.5 <[email protected]>
• f02aed40e495f24ad09576bcb9384fefb152d7f2 on 14/12/2025 2:12 pm by Zohaib Sibte Hassan <[email protected]> [GitHub Diff]
  Message: Merge feature/grouped-2pc-v2: StatementCode consolidation and security fixes
• 9b9cfb3c15caf47232d34db7bad5796f14450dc4 on 14/12/2025 2:11 pm by Zohaib Sibte Hassan <[email protected]> [GitHub Diff]
  Message: Fix code review issues: error handling and hot path optimization
  
  - Return error instead of panic in convertStatementsToProto for unknown types
  - Guard debug logs in sendChangeEvent with Enabled() check to avoid
  string formatting overhead in streaming hot path
  - Fix readLengthEncodedInt to return (0,0) on truncated buffer for
  consistent failure signaling
  
  ???? Generated with [Claude Code](https://claude.com/claude-code)
  
  Co-Authored-By: Claude Opus 4.5 <[email protected]>
• e58f2c60db3a2d3cc66bbd8294cb385f89430d5f on 14/12/2025 1:10 pm by Zohaib Sibte Hassan <[email protected]> [GitHub Diff]
  Message: Consolidate StatementCode in common package with strict wire boundary
  
  - Create common/ package as single source of truth for StatementCode
  - Move StatementType proto to grpc/common/ to avoid import cycles
  - Add map-based converters with panic on unknown types
  - Enforce strict boundary: internal StatementCode vs wire StatementType
  - Remove redundant type definitions from protocol/ and query/
  - All 1,113 unit tests and 29 integration tests pass
  
  ???? Generated with [Claude Code](https://claude.com/claude-code)
  
  Co-Authored-By: Claude Opus 4.5 <[email protected]>
• 8f90f5a20c338bb67cc361361cf8cb4d8ae19464 on 14/12/2025 11:40 am by Zohaib Sibte Hassan <[email protected]> [GitHub Diff]
  Message: Fix prepared statement string params for SQLite TEXT comparison
  
  Return text-type MySQL parameters (STRING, VARCHAR, etc.) as Go
  strings instead of []byte. SQLite TEXT column comparisons fail when
  comparing against BLOB-typed []byte values, causing queries like
  "SELECT name FROM sqlite_master WHERE name = ?" to return no rows
  even when the table exists.
  
  This fixes TestClusterReplication which was failing because the
  table name parameter was passed as []byte.
  
  ???? Generated with [Claude Code](https://claude.com/claude-code)
  
  Co-Authored-By: Claude Opus 4.5 <[email protected]>
• 3a85ca5c970c87119d88fce406a1ce1985804c1a on 14/12/2025 11:19 am by Zohaib Sibte Hassan <[email protected]> [GitHub Diff]
  Message: Rename StatementType to StatementCode for type safety
  
  - Rename protocol.StatementType to protocol.StatementCode to
  differentiate from grpc.StatementType (proto-generated)
  - Add map-based converters in grpc/statement_code_map.go
  - Fix direct cast bug at grpc/replication_handler.go:979
  - Add comprehensive tests for conversion functions
  
  The two types have different values (protocol uses iota, grpc uses
  protobuf values). Using different names prevents accidental mixing.
  
  ???? Generated with [Claude Code](https://claude.com/claude-code)
  
  Co-Authored-By: Claude Opus 4.5 <[email protected]>
• 54a9a0a3d54299e4abb27e32c1b4f9421e558638 on 14/12/2025 8:51 am by Zohaib Sibte Hassan <[email protected]> [GitHub Diff]
  Message: Fix SQL injection vulnerability by removing string interpolation
  
  SECURITY: Remove buildQueryWithParams() which did dangerous string
  interpolation of user-provided values before CDC hooks fired.
  
  Changes:
  - Add ExecutionRequest type to coordinator for local execution params
  - Update ConnectionHandler.HandleQuery to accept params []interface{}
  - Thread params through CoordinatorHandler and ReadOnlyHandler
  - Update db.ExecContext to use variadic args for prepared statements
  - Delete vulnerable functions: buildQueryWithParams, formatParam, escapeString
  
  Architecture:
  - Wire protocol params now passed directly to SQLite prepared statements
  - DML uses row data (OldValues/NewValues) for CDC, not SQL - already safe
  - DDL uses raw SQL (no user params) - already safe
  - All replica replay paths verified to use parameterized queries
  
  Tests:
  - Add parameterized_query_test.go with SQL injection prevention tests
  - Add CDC contract tests verifying Statement has no Params field
  - Update existing tests to use new HandleQuery signature
  
  ???? Generated with [Claude Code](https://claude.com/claude-code)
  
  Co-Authored-By: Claude Opus 4.5 <[email protected]>
• 43ee8a3a2a09fd01793a39035bb9b6f1b25b9176 on 14/12/2025 7:26 am by Zohaib Sibte Hassan <[email protected]> [GitHub Diff]
  Message: Add cluster integration test and schema cache error handling
  
  - Add 3-node cluster integration test verifying DDL/DML replication
  - Add ErrSchemaCacheMiss error type for distinguishable schema errors
  - Add retry logic in CDC replay to reload schema on cache miss
  - Fix catch-up logic to check for actual data (non-zero txn IDs)
  - Fix database import to run when catch-up not needed or failed
  - Fix shouldReplaceWithImport to handle empty vs populated databases
  
  ???? Generated with [Claude Code](https://claude.com/claude-code)
  
  Co-Authored-By: Claude Opus 4.5 <[email protected]>
• 91a5941a77ed585e25afa3bfa8f2ff769edf93fd on 13/12/2025 11:02 am by Zohaib Sibte Hassan <[email protected]> [GitHub Diff]
  Message: Optimize hot path logging and use cached schema for CDC operations
  
  Hot path logging cleanup:
  - Remove debug logs from applyDataChanges, rebuildStatementsFromCDC
  - Remove per-entry loop logging in rebuildStatementsFromCDC (was O(n) logs)
  - Remove "Transaction still active" debug log from GC scan
  - Remove AcquireCDCRowLock conflict debug log (error return suffices)
  
  Error handling improvements:
  - Replace silent `_ = s.Heartbeat(txnID)` with Warn-level error logging
  - Heartbeat failures now logged in WriteIntent and WriteIntentEntry
  
  Schema caching optimization:
  - Add GetCachedTableSchema() to ReplicatedDatabase for in-memory lookups
  - Switch delta_sync applyCDCUpdate/Delete to use cached schema
  - Switch replication_handler applyReplayCDCUpdate/Delete to cached schema
  - Eliminates SQLite queries during CDC replay operations
  
  DDL replication fix:
  - Increment schema version on followers after DDL commit replication
  - Ensures follower schema cache stays in sync with leader
  
  ???? Generated with [Claude Code](https://claude.com/claude-code)
  
  Co-Authored-By: Claude Opus 4.5 <[email protected]>