Community Contributors List
Congratulations to @lifepuzzlefun for becoming one of our active contributors.
- @lifepuzzlefun：#1411、#1447、#1469、#1487、#1493、#1507、#1513、#1518、#1521、#1522、#1535
Welcome @yx9o, @jitokim, @wensongz, @vintagewang, @sullis, @szepeviktor, @llzcx, and @mapan1984 for bringing fresh energy into the community and successfully merging your first PR.
AutoMQ Trunk Updates
Direct on S3 Early Access
https://github.com/AutoMQ/automq/releases/tag/1.2.0-beta0
The Direct on S3 Early Access version is now available! Experience diskless management, 99.999999999% data durability, and "zero-overhead" AZ-level disaster recovery with just a one-line configuration change.
s3.wal.path=0@s3://<your-bucket>?region=<your-region>&endpoint=<your-s3-endpoint>
100k Partitions, 10GiB/s Large-Scale Cluster Metadata Optimization
#1419 #1464
Replaced S3Object and S3StreamMetadata memory data structures in Delta and Image with TimelineHashMap to avoid frequent metadata changes triggering Map copies.
Offloaded StreamSetObject memory index information to object storage, reducing KRaft metadata memory usage and checkpoint space usage.
Optimization results: AutoMQ can stably operate under 100k active write partitions, 7GiB/s write throughput, and 90 nodes with 2C16G configuration. KRaft checkpoint space usage reduced from 600MiB to 100MiB.
New Composite Object Type
#1432 #1437 #1443
The Composite Object logically combines multiple physical objects into a larger one through symbolic links. The individual physical objects are typically around 4MB in size, while the Composite Object can logically link up to 10 GiB of data.
Compaction based on Composite Objects eliminates the need to create MultiPart Objects and use the UploadPartCopy API, leading to faster merge speeds and reduced merge costs.
Optimization results: The cost of invoking the Compaction API and the time required for compaction are only 10% of those of the original physical merge.