We are excited to announce the release of Alluxio v1.1. This release greatly improves the performance and usability of Alluxio. Highlights of major features and updates are listed below.
Downloads for the release can be found here: Downloads. We will also have a meetup at Intel on June 15th to provide present updates regarding this release as well.
Performance Improvements
- Improved performance of metadata operations: Alluxio 1.1 master utilizes multiple CPUs more effectively, achieving up to 20x higher throughput and up to 5x lower 99 percentile latency for metadata heavy workloads.
- Improved performance of reading Parquet files: Workloads with heavy random I/Os, such as reading Parquet files in Spark, are better supported, In particular, Alluxio now allows users to cache partially read blocks with higher read concurrency.
- Improved performance of writing small files: By optimizing storage structures, Alluxio 1.1 achieves up to 1000x higher throughput when writing small files.
Usability Improvements
- Simplified under storage metadata-loading process: Alluxio 1.1 greatly simplifies the work for users to surface under storage information in Alluxio. Loading information about files and directories from under storage now happens automatically the first time a file or directory is accessed.
- Easy configuration: Alluxio configurations, such as conf/alluxio-env.sh, have been restructured to be simpler to reason about and maintain. Alluxio also provides easier ways to customize Alluxio properties for external jobs (e.g., Spark or MapReduce) which interact with Alluxio. These changes help admins who launch and maintain Alluxio as well as users who run applications on Alluxio.
- Deployment without Sudo Access: To help users try out Alluxio, from the 1.1 release onward, Alluxio can be deployed without sudo access. This feature is targeted toward users who want to try out the system’s features but do no require performance guarantees.
- Under storage I/O delegation (Alpha): To simplify application development and maintenance, Alluxio 1.1 introduces an experimental option to read and write data from the under storage system through Alluxio workers. For example, applications will no longer require libraries to interact with under storage systems, greatly reducing the complexity to develop, configure, and maintain applications. This feature can be enabled by setting the flag alluxio.user.ufs.delegation.enabled=true and is disabled by default.
- Improved user and file system permission checking: Alluxio provides permission semantics similar to the standard UNIX permission semantics, with command-line interfaces including chmod, chown, and chgrp. This feature can be enabled by setting the flag alluxio.security.authorization.permission.enabled=true and is disabled by default.
New Integrations
- Integration with GCS and GCE: Alluxio supports using Google Cloud Storage (GCS) as an under storage system. Users can use the one-click Alluxio deployment to launch a cluster on Google Compute Engine (GCE) which is backed by GCS as the under storage.
Acknowledgements
We would like to thank the following members of the community for their contributions to Alluxio 1.1. The release would not have been possible without your efforts! Alex Naspo, Andrew Audibert, Andy Robb, Archer Liu, Austin Walters, Bin Fan, Brennon York, Cagdas Senol, Calvin Jia, Carlos de Carvalho, Chang Cheng, Chao Sun, Chaomin Yu, Chen Tian, Chendai Rui, Daniel Yu, Dehong Song, Dong Chen, Dongming Liang, Eric Anderson, Ewout Van Gossum, Gene Pang, Gil Vernik, Grace Huang, Guanghui Zhu, Hailin Wang, Haoyuan Li, Hechen Gao, Ilya Ganelin, Jack Kosaian, Jan Hentschel, Jiateng Wang, Jiri Simsa, Kaixuan Huang, Lei Qi, Lei(Brick) Xu, Leo Wang, Li Luo, Lin Dong, Linghton, Meng Tao, Menglei Sun, Mingfei Shi, MrHohn, Ngyuen Xuan, Nirmal Ranganathan, Nolan Semrau, Pei Sun, Pengfei Xuan, QianHao Dong, Ray Zhang, Rico Chiu, Rong Gu, Ruichen Dai, Ryan Walton, Sandeep Narayanaswami, Saverio Veltri, Shaohui Liu, Shaoshan Liu, Shengbin Huang, Shumin, Vijay Bhat, Wesley Tang, Xiakun Lu, Xianyang Liu, Xiao Qian, Xinxin Huang, Yangming Xie, Yu Siyang, Yuanqing Ye, Yufa Zhou, Yun Li, Yupeng Fu, Zhi Huang, liuweialberic, qzhong