|
|
@@ -4,24 +4,22 @@
|
|
|
[](https://travis-ci.org/chrislusf/seaweedfs)
|
|
|
[](https://godoc.org/github.com/chrislusf/seaweedfs/weed)
|
|
|
[](https://github.com/chrislusf/seaweedfs/wiki)
|
|
|
-[](https://hub.docker.com/r/chrislusf/seaweedfs/)
|
|
|
+[](https://hub.docker.com/r/chrislusf/seaweedfs/)
|
|
|
|
|
|
|
|
|
|
|
|
<h2 align="center">Supporting SeaweedFS</h2>
|
|
|
|
|
|
-SeaweedFS is an independent Apache-licensed open source project with its ongoing development made
|
|
|
-possible entirely thanks to the support of these awesome [backers](https://github.com/chrislusf/seaweedfs/blob/master/backers.md).
|
|
|
+SeaweedFS is an independent Apache-licensed open source project with its ongoing development made
|
|
|
+possible entirely thanks to the support of these awesome [backers](https://github.com/chrislusf/seaweedfs/blob/master/backers.md).
|
|
|
If you'd like to grow SeaweedFS even stronger, please consider joining our
|
|
|
<a href="https://www.patreon.com/seaweedfs">sponsors on Patreon</a>.
|
|
|
|
|
|
-Platinum ($2500/month), Gold ($500/month): put your company logo on the SeaweedFS github page
|
|
|
-Generous Backer($50/month), Backer($10/month): put your name on the SeaweedFS backer page.
|
|
|
-
|
|
|
Your support will be really appreciated by me and other supporters!
|
|
|
|
|
|
<h3 align="center"><a href="https://www.patreon.com/seaweedfs">Sponsor SeaweedFS via Patreon</a></h3>
|
|
|
|
|
|
+<!--
|
|
|
<h4 align="center">Platinum</h4>
|
|
|
|
|
|
<p align="center">
|
|
|
@@ -45,61 +43,108 @@ Your support will be really appreciated by me and other supporters!
|
|
|
</tbody>
|
|
|
</table>
|
|
|
|
|
|
+-->
|
|
|
+
|
|
|
---
|
|
|
|
|
|
|
|
|
- [Download Binaries for different platforms](https://github.com/chrislusf/seaweedfs/releases/latest)
|
|
|
-- [SeaweedFS on Slack](https://join.slack.com/t/seaweedfs/shared_invite/enQtMzI4MTMwMjU2MzA3LTc4MmVlYmFlNjBmZTgzZmJlYmI1MDE1YzkyNWYyZjkwZDFiM2RlMDdjNjVlNjdjYzc4NGFhZGIyYzEyMzJkYTA)
|
|
|
+- [SeaweedFS on Slack](https://join.slack.com/t/seaweedfs/shared_invite/enQtMzI4MTMwMjU2MzA3LTEyYzZmZWYzOGQ3MDJlZWMzYmI0OTE4OTJiZjJjODBmMzUxNmYwODg0YjY3MTNlMjBmZDQ1NzQ5NDJhZWI2ZmY)
|
|
|
+- [SeaweedFS on Twitter](https://twitter.com/SeaweedFS)
|
|
|
- [SeaweedFS Mailing List](https://groups.google.com/d/forum/seaweedfs)
|
|
|
- [Wiki Documentation](https://github.com/chrislusf/seaweedfs/wiki)
|
|
|
-
|
|
|
-
|
|
|
-## Introduction
|
|
|
+- [SeaweedFS Introduction Slides](https://www.slideshare.net/chrislusf/seaweedfs-introduction)
|
|
|
+
|
|
|
+Table of Contents
|
|
|
+=================
|
|
|
+
|
|
|
+* [Introduction](#introduction)
|
|
|
+* [Features](#features)
|
|
|
+ * [Additional Features](#additional-features)
|
|
|
+ * [Filer Features](#filer-features)
|
|
|
+* [Example Usage](#example-usage)
|
|
|
+* [Architecture](#architecture)
|
|
|
+* [Compared to Other File Systems](#compared-to-other-file-systems)
|
|
|
+ * [Compared to HDFS](#compared-to-hdfs)
|
|
|
+ * [Compared to GlusterFS, Ceph](#compared-to-glusterfs-ceph)
|
|
|
+ * [Compared to GlusterFS](#compared-to-glusterfs)
|
|
|
+ * [Compared to Ceph](#compared-to-ceph)
|
|
|
+* [Dev Plan](#dev-plan)
|
|
|
+* [Installation Guide](#installation-guide)
|
|
|
+* [Disk Related Topics](#disk-related-topics)
|
|
|
+* [Benchmark](#Benchmark)
|
|
|
+* [License](#license)
|
|
|
+
|
|
|
+## Introduction ##
|
|
|
|
|
|
SeaweedFS is a simple and highly scalable distributed file system. There are two objectives:
|
|
|
|
|
|
1. to store billions of files!
|
|
|
2. to serve the files fast!
|
|
|
|
|
|
-SeaweedFS started as an Object Store to handle small files efficiently. Instead of managing all file metadata in a central master, the central master only manages file volumes, and it lets these volume servers manage files and their metadata. This relieves concurrency pressure from the central master and spreads file metadata into volume servers, allowing faster file access (just one disk read operation).
|
|
|
+SeaweedFS started as an Object Store to handle small files efficiently. Instead of managing all file metadata in a central master, the central master only manages file volumes, and it lets these volume servers manage files and their metadata. This relieves concurrency pressure from the central master and spreads file metadata into volume servers, allowing faster file access (O(1), usually just one disk read operation).
|
|
|
+
|
|
|
+SeaweedFS can transparently integrate with the cloud. With hot data on local cluster, and warm data on the cloud with O(1) access time, SeaweedFS can achieve both fast local access time and elastic cloud storage capacity, without any client side changes.
|
|
|
|
|
|
There is only 40 bytes of disk storage overhead for each file's metadata. It is so simple with O(1) disk reads that you are welcome to challenge the performance with your actual use cases.
|
|
|
|
|
|
-SeaweedFS started by implementing [Facebook's Haystack design paper](http://www.usenix.org/event/osdi10/tech/full_papers/Beaver.pdf).
|
|
|
+SeaweedFS started by implementing [Facebook's Haystack design paper](http://www.usenix.org/event/osdi10/tech/full_papers/Beaver.pdf). Also, SeaweedFS implements erasure coding with ideas from [f4: Facebook’s Warm BLOB Storage System](https://www.usenix.org/system/files/conference/osdi14/osdi14-paper-muralidhar.pdf)
|
|
|
|
|
|
-SeaweedFS can work very well with just the object store. [[Filer]] can then be added later to support directories and POSIX attributes. Filer is a separate linearly-scalable stateless server with customizable metadata stores, e.g., MySql/Postgres/Redis/Cassandra/LevelDB.
|
|
|
+On top of the object store, optional [Filer] can support directories and POSIX attributes. Filer is a separate linearly-scalable stateless server with customizable metadata stores, e.g., MySql, Postgres, Mongodb, Redis, Etcd, Cassandra, LevelDB, MemSql, TiDB, TiKV, CockroachDB, etc.
|
|
|
|
|
|
-## Additional Features
|
|
|
-* Can choose no replication or different replication levels, rack and data center aware
|
|
|
-* Automatic master servers failover - no single point of failure (SPOF)
|
|
|
-* Automatic Gzip compression depending on file mime type
|
|
|
-* Automatic compaction to reclaim disk space after deletion or update
|
|
|
-* Servers in the same cluster can have different disk spaces, file systems, OS etc.
|
|
|
-* Adding/Removing servers does **not** cause any data re-balancing
|
|
|
-* Optionally fix the orientation for jpeg pictures
|
|
|
-* Support Etag, Accept-Range, Last-Modified, etc.
|
|
|
-* Support in-memory/leveldb/boltdb/btree mode tuning for memory/performance balance.
|
|
|
+[Back to TOC](#table-of-contents)
|
|
|
|
|
|
-## Filer Features
|
|
|
+## Additional Features ##
|
|
|
+* Can choose no replication or different replication levels, rack and data center aware.
|
|
|
+* Automatic master servers failover - no single point of failure (SPOF).
|
|
|
+* Automatic Gzip compression depending on file mime type.
|
|
|
+* Automatic compaction to reclaim disk space after deletion or update.
|
|
|
+* [Automatic entry TTL expiration][VolumeServerTTL].
|
|
|
+* Any server with some disk spaces can add to the total storage space.
|
|
|
+* Adding/Removing servers does **not** cause any data re-balancing.
|
|
|
+* Optional picture resizing.
|
|
|
+* Support ETag, Accept-Range, Last-Modified, etc.
|
|
|
+* Support in-memory/leveldb/readonly mode tuning for memory/performance balance.
|
|
|
+* Support rebalancing the writable and readonly volumes.
|
|
|
+* [Transparent cloud integration][CloudTier]: unlimited capacity via tiered cloud storage for warm data.
|
|
|
+* [Erasure Coding for warm storage][ErasureCoding] Rack-Aware 10.4 erasure coding reduces storage cost and increases availability.
|
|
|
+
|
|
|
+[Back to TOC](#table-of-contents)
|
|
|
+
|
|
|
+## Filer Features ##
|
|
|
* [filer server][Filer] provide "normal" directories and files via http.
|
|
|
* [mount filer][Mount] to read and write files directly as a local directory via FUSE.
|
|
|
* [Amazon S3 compatible API][AmazonS3API] to access files with S3 tooling.
|
|
|
* [Hadoop Compatible File System][Hadoop] to access files from Hadoop/Spark/Flink/etc jobs.
|
|
|
* [Async Backup To Cloud][BackupToCloud] has extremely fast local access and backups to Amazon S3, Google Cloud Storage, Azure, BackBlaze.
|
|
|
+* [WebDAV] access as a mapped drive on Mac and Windows, or from mobile devices.
|
|
|
+* [AES256-GCM Encrypted Storage][FilerDataEncryption] safely stores the encrypted data.
|
|
|
+* [File TTL][FilerTTL] automatically purge file metadata and actual file data.
|
|
|
+* [Kubernetes CSI Driver][SeaweedFsCsiDriver] A Container Storage Interface (CSI) Driver. [](https://hub.docker.com/r/chrislusf/seaweedfs-csi-driver/)
|
|
|
|
|
|
[Filer]: https://github.com/chrislusf/seaweedfs/wiki/Directories-and-Files
|
|
|
[Mount]: https://github.com/chrislusf/seaweedfs/wiki/Mount
|
|
|
[AmazonS3API]: https://github.com/chrislusf/seaweedfs/wiki/Amazon-S3-API
|
|
|
[BackupToCloud]: https://github.com/chrislusf/seaweedfs/wiki/Backup-to-Cloud
|
|
|
[Hadoop]: https://github.com/chrislusf/seaweedfs/wiki/Hadoop-Compatible-File-System
|
|
|
+[WebDAV]: https://github.com/chrislusf/seaweedfs/wiki/WebDAV
|
|
|
+[ErasureCoding]: https://github.com/chrislusf/seaweedfs/wiki/Erasure-coding-for-warm-storage
|
|
|
+[CloudTier]: https://github.com/chrislusf/seaweedfs/wiki/Cloud-Tier
|
|
|
+[FilerDataEncryption]: https://github.com/chrislusf/seaweedfs/wiki/Filer-Data-Encryption
|
|
|
+[FilerTTL]: https://github.com/chrislusf/seaweedfs/wiki/Filer-Stores
|
|
|
+[VolumeServerTTL]: https://github.com/chrislusf/seaweedfs/wiki/Store-file-with-a-Time-To-Live
|
|
|
+[SeaweedFsCsiDriver]: https://github.com/seaweedfs/seaweedfs-csi-driver
|
|
|
+
|
|
|
+[Back to TOC](#table-of-contents)
|
|
|
+
|
|
|
+## Example Usage ##
|
|
|
|
|
|
-## Example Usage
|
|
|
By default, the master node runs on port 9333, and the volume nodes run on port 8080.
|
|
|
Let's start one master node, and two volume nodes on port 8080 and 8081. Ideally, they should be started from different machines. We'll use localhost as an example.
|
|
|
|
|
|
SeaweedFS uses HTTP REST operations to read, write, and delete. The responses are in JSON or JSONP format.
|
|
|
|
|
|
-### Start Master Server
|
|
|
+### Start Master Server ###
|
|
|
|
|
|
```
|
|
|
> ./weed master
|
|
|
@@ -125,7 +170,7 @@ Second, to store the file content, send a HTTP multi-part POST request to `url +
|
|
|
|
|
|
```
|
|
|
> curl -F file=@/home/chris/myphoto.jpg http://127.0.0.1:8080/3,01637037d6
|
|
|
-{"size": 43234}
|
|
|
+{"name":"myphoto.jpg","size":43234,"eTag":"1cc0118e"}
|
|
|
```
|
|
|
|
|
|
To update, send another POST request with updated file content.
|
|
|
@@ -135,6 +180,7 @@ For deletion, send an HTTP DELETE request to the same `url + '/' + fid` URL:
|
|
|
```
|
|
|
> curl -X DELETE http://127.0.0.1:8080/3,01637037d6
|
|
|
```
|
|
|
+
|
|
|
### Save File Id ###
|
|
|
|
|
|
Now, you can save the `fid`, 3,01637037d6 in this case, to a database field.
|
|
|
@@ -157,7 +203,7 @@ First look up the volume server's URLs by the file's volumeId:
|
|
|
|
|
|
```
|
|
|
> curl http://localhost:9333/dir/lookup?volumeId=3
|
|
|
-{"locations":[{"publicUrl":"localhost:8080","url":"localhost:8080"}]}
|
|
|
+{"volumeId":"3","locations":[{"publicUrl":"localhost:8080","url":"localhost:8080"}]}
|
|
|
```
|
|
|
|
|
|
Since (usually) there are not too many volume servers, and volumes don't move often, you can cache the results most of the time. Depending on the replication type, one volume can have multiple replica locations. Just randomly pick one location to read.
|
|
|
@@ -213,7 +259,7 @@ More details about replication can be found [on the wiki][Replication].
|
|
|
|
|
|
You can also set the default replication strategy when starting the master server.
|
|
|
|
|
|
-### Allocate File Key on specific data center ###
|
|
|
+### Allocate File Key on Specific Data Center ###
|
|
|
|
|
|
Volume servers can be started with a specific data center name:
|
|
|
|
|
|
@@ -239,6 +285,8 @@ When requesting a file key, an optional "dataCenter" parameter can limit the ass
|
|
|
[feat-3]: https://github.com/chrislusf/seaweedfs/wiki/Optimization#upload-large-files
|
|
|
[feat-4]: https://github.com/chrislusf/seaweedfs/wiki/Optimization#collection-as-a-simple-name-space
|
|
|
|
|
|
+[Back to TOC](#table-of-contents)
|
|
|
+
|
|
|
## Architecture ##
|
|
|
|
|
|
Usually distributed file systems split each file into chunks, a central master keeps a mapping of filenames, chunk indices to chunk handles, and also which chunks each chunk server has.
|
|
|
@@ -279,12 +327,26 @@ Each individual file size is limited to the volume size.
|
|
|
|
|
|
All file meta information stored on an volume server is readable from memory without disk access. Each file takes just a 16-byte map entry of <64bit key, 32bit offset, 32bit size>. Of course, each map entry has its own space cost for the map. But usually the disk space runs out before the memory does.
|
|
|
|
|
|
+### Tiered Storage to the cloud ###
|
|
|
+
|
|
|
+The local volume servers are much faster, while cloud storages have elastic capacity and are actually more cost-efficient if not accessed often (usually free to upload, but relatively costly to access). With the append-only structure and O(1) access time, SeaweedFS can take advantage of both local and cloud storage by offloading the warm data to the cloud.
|
|
|
+
|
|
|
+Usually hot data are fresh and warm data are old. SeaweedFS puts the newly created volumes on local servers, and optionally upload the older volumes on the cloud. If the older data are accessed less often, this literally gives you unlimited capacity with limited local servers, and still fast for new data.
|
|
|
+
|
|
|
+With the O(1) access time, the network latency cost is kept at minimum.
|
|
|
+
|
|
|
+If the hot~warm data is split as 20~80, with 20 servers, you can achieve storage capacity of 100 servers. That's a cost saving of 80%! Or you can repurpose the 80 servers to store new data also, and get 5X storage throughput.
|
|
|
+
|
|
|
+[Back to TOC](#table-of-contents)
|
|
|
+
|
|
|
## Compared to Other File Systems ##
|
|
|
|
|
|
Most other distributed file systems seem more complicated than necessary.
|
|
|
|
|
|
SeaweedFS is meant to be fast and simple, in both setup and operation. If you do not understand how it works when you reach here, we've failed! Please raise an issue with any questions or update this file with clarifications.
|
|
|
|
|
|
+[Back to TOC](#table-of-contents)
|
|
|
+
|
|
|
### Compared to HDFS ###
|
|
|
|
|
|
HDFS uses the chunk approach for each file, and is ideal for storing large files.
|
|
|
@@ -293,6 +355,7 @@ SeaweedFS is ideal for serving relatively smaller files quickly and concurrently
|
|
|
|
|
|
SeaweedFS can also store extra large files by splitting them into manageable data chunks, and store the file ids of the data chunks into a meta chunk. This is managed by "weed upload/download" tool, and the weed master or volume servers are agnostic about it.
|
|
|
|
|
|
+[Back to TOC](#table-of-contents)
|
|
|
|
|
|
### Compared to GlusterFS, Ceph ###
|
|
|
|
|
|
@@ -300,7 +363,7 @@ The architectures are mostly the same. SeaweedFS aims to store and read files fa
|
|
|
|
|
|
* SeaweedFS optimizes for small files, ensuring O(1) disk seek operation, and can also handle large files.
|
|
|
* SeaweedFS statically assigns a volume id for a file. Locating file content becomes just a lookup of the volume id, which can be easily cached.
|
|
|
-* SeaweedFS Filer metadata store can be any well-known and proven data stores, e.g., Cassandra, Redis, MySql, Postgres, etc, and is easy to customized.
|
|
|
+* SeaweedFS Filer metadata store can be any well-known and proven data stores, e.g., Cassandra, Mongodb, Redis, Etcd, MySql, Postgres, MemSql, TiDB, CockroachDB, etc, and is easy to customized.
|
|
|
* SeaweedFS Volume server also communicates directly with clients via HTTP, supporting range queries, direct uploads, etc.
|
|
|
|
|
|
| System | File Meta | File Content Read| POSIX | REST API | Optimized for small files |
|
|
|
@@ -309,6 +372,9 @@ The architectures are mostly the same. SeaweedFS aims to store and read files fa
|
|
|
| SeaweedFS Filer| Linearly Scalable, Customizable | O(1) disk seek | FUSE | Yes | Yes |
|
|
|
| GlusterFS | hashing | | FUSE, NFS | | |
|
|
|
| Ceph | hashing + rules | | FUSE | Yes | |
|
|
|
+| MooseFS | in memory | | FUSE | | No |
|
|
|
+
|
|
|
+[Back to TOC](#table-of-contents)
|
|
|
|
|
|
### Compared to GlusterFS ###
|
|
|
|
|
|
@@ -316,11 +382,21 @@ GlusterFS stores files, both directories and content, in configurable volumes ca
|
|
|
|
|
|
GlusterFS hashes the path and filename into ids, and assigned to virtual volumes, and then mapped to "bricks".
|
|
|
|
|
|
+[Back to TOC](#table-of-contents)
|
|
|
+
|
|
|
+### Compared to MooseFS ###
|
|
|
+
|
|
|
+MooseFS chooses to neglect small file issue. From moosefs 3.0 manual, "even a small file will occupy 64KiB plus additionally 4KiB of checksums and 1KiB for the header", because it "was initially designed for keeping large amounts (like several thousands) of very big files"
|
|
|
+
|
|
|
+MooseFS Master Server keeps all meta data in memory. Same issue as HDFS namenode.
|
|
|
+
|
|
|
+[Back to TOC](#table-of-contents)
|
|
|
+
|
|
|
### Compared to Ceph ###
|
|
|
|
|
|
Ceph can be setup similar to SeaweedFS as a key->blob store. It is much more complicated, with the need to support layers on top of it. [Here is a more detailed comparison](https://github.com/chrislusf/seaweedfs/issues/120)
|
|
|
|
|
|
-SeaweedFS has a centralized master group to look up free volumes, while Ceph uses hashing and metadata servers to locate its objects. Having a centralized master makes it easy to code and manage.
|
|
|
+SeaweedFS has a centralized master group to look up free volumes, while Ceph uses hashing and metadata servers to locate its objects. Having a centralized master makes it easy to code and manage.
|
|
|
|
|
|
Same as SeaweedFS, Ceph is also based on the object store RADOS. Ceph is rather complicated with mixed reviews.
|
|
|
|
|
|
@@ -328,7 +404,7 @@ Ceph uses CRUSH hashing to automatically manage the data placement. SeaweedFS pl
|
|
|
|
|
|
SeaweedFS is optimized for small files. Small files are stored as one continuous block of content, with at most 8 unused bytes between files. Small file access is O(1) disk read.
|
|
|
|
|
|
-SeaweedFS Filer uses off-the-shelf stores, such as MySql, Postgres, Redis, Cassandra, to manage file directories. There are proven, scalable, and easier to manage.
|
|
|
+SeaweedFS Filer uses off-the-shelf stores, such as MySql, Postgres, Mongodb, Redis, Etcd, Cassandra, MemSql, TiDB, CockroachCB, to manage file directories. These stores are proven, scalable, and easier to manage.
|
|
|
|
|
|
| SeaweedFS | comparable to Ceph | advantage |
|
|
|
| ------------- | ------------- | ---------------- |
|
|
|
@@ -336,16 +412,26 @@ SeaweedFS Filer uses off-the-shelf stores, such as MySql, Postgres, Redis, Cassa
|
|
|
| Volume | OSD | optimized for small files |
|
|
|
| Filer | Ceph FS | linearly scalable, Customizable, O(1) or O(logN) |
|
|
|
|
|
|
+[Back to TOC](#table-of-contents)
|
|
|
|
|
|
-## Dev plan ##
|
|
|
+## Dev Plan ##
|
|
|
|
|
|
More tools and documentation, on how to maintain and scale the system. For example, how to move volumes, automatically balancing data, how to grow volumes, how to check system status, etc.
|
|
|
Other key features include: Erasure Encoding, JWT security.
|
|
|
|
|
|
This is a super exciting project! And we need helpers and [support](https://www.patreon.com/seaweedfs)!
|
|
|
|
|
|
+BTW, We suggest run the code style check script `util/gostd` before you push your branch to remote, it will make SeaweedFS easy to review, maintain and develop:
|
|
|
+
|
|
|
+```
|
|
|
+$ ./util/gostd
|
|
|
+```
|
|
|
|
|
|
-## Installation guide for users who are not familiar with golang
|
|
|
+[Back to TOC](#table-of-contents)
|
|
|
+
|
|
|
+## Installation Guide ##
|
|
|
+
|
|
|
+> Installation guide for users who are not familiar with golang
|
|
|
|
|
|
Step 1: install go on your machine and setup the environment by following the instructions at:
|
|
|
|
|
|
@@ -366,77 +452,87 @@ go get github.com/chrislusf/seaweedfs/weed
|
|
|
|
|
|
Once this is done, you will find the executable "weed" in your `$GOPATH/bin` directory
|
|
|
|
|
|
-Step 4: after you modify your code locally, you could start a local build by calling `go install` under
|
|
|
+Note:
|
|
|
+* If you got into this problem, try to `rm -Rf $GOPATH/src/go.etcd.io/etcd/vendor/golang.org/x/net/trace` and build again.
|
|
|
+```
|
|
|
+panic: /debug/requests is already registered. You may have two independent copies of golang.org/x/net/trace in your binary, trying to maintain separate state. This may involve a vendored copy of golang.org/x/net/trace.
|
|
|
+```
|
|
|
+
|
|
|
+Step 4: after you modify your code locally, you could start a local build by calling `go install` under
|
|
|
|
|
|
```
|
|
|
$GOPATH/src/github.com/chrislusf/seaweedfs/weed
|
|
|
```
|
|
|
|
|
|
-## Disk Related topics ##
|
|
|
+[Back to TOC](#table-of-contents)
|
|
|
+
|
|
|
+## Disk Related Topics ##
|
|
|
|
|
|
### Hard Drive Performance ###
|
|
|
|
|
|
When testing read performance on SeaweedFS, it basically becomes a performance test of your hard drive's random read speed. Hard drives usually get 100MB/s~200MB/s.
|
|
|
|
|
|
-### Solid State Disk
|
|
|
+### Solid State Disk ###
|
|
|
|
|
|
To modify or delete small files, SSD must delete a whole block at a time, and move content in existing blocks to a new block. SSD is fast when brand new, but will get fragmented over time and you have to garbage collect, compacting blocks. SeaweedFS is friendly to SSD since it is append-only. Deletion and compaction are done on volume level in the background, not slowing reading and not causing fragmentation.
|
|
|
|
|
|
-## Benchmark
|
|
|
+[Back to TOC](#table-of-contents)
|
|
|
+
|
|
|
+## Benchmark ##
|
|
|
|
|
|
My Own Unscientific Single Machine Results on Mac Book with Solid State Disk, CPU: 1 Intel Core i7 2.6GHz.
|
|
|
|
|
|
Write 1 million 1KB file:
|
|
|
```
|
|
|
Concurrency Level: 16
|
|
|
-Time taken for tests: 88.796 seconds
|
|
|
+Time taken for tests: 66.753 seconds
|
|
|
Complete requests: 1048576
|
|
|
Failed requests: 0
|
|
|
-Total transferred: 1106764659 bytes
|
|
|
-Requests per second: 11808.87 [#/sec]
|
|
|
-Transfer rate: 12172.05 [Kbytes/sec]
|
|
|
+Total transferred: 1106789009 bytes
|
|
|
+Requests per second: 15708.23 [#/sec]
|
|
|
+Transfer rate: 16191.69 [Kbytes/sec]
|
|
|
|
|
|
Connection Times (ms)
|
|
|
min avg max std
|
|
|
-Total: 0.2 1.3 44.8 0.9
|
|
|
+Total: 0.3 1.0 84.3 0.9
|
|
|
|
|
|
Percentage of the requests served within a certain time (ms)
|
|
|
- 50% 1.1 ms
|
|
|
- 66% 1.3 ms
|
|
|
- 75% 1.5 ms
|
|
|
- 80% 1.7 ms
|
|
|
- 90% 2.1 ms
|
|
|
- 95% 2.6 ms
|
|
|
- 98% 3.7 ms
|
|
|
- 99% 4.6 ms
|
|
|
- 100% 44.8 ms
|
|
|
+ 50% 0.8 ms
|
|
|
+ 66% 1.0 ms
|
|
|
+ 75% 1.1 ms
|
|
|
+ 80% 1.2 ms
|
|
|
+ 90% 1.4 ms
|
|
|
+ 95% 1.7 ms
|
|
|
+ 98% 2.1 ms
|
|
|
+ 99% 2.6 ms
|
|
|
+ 100% 84.3 ms
|
|
|
```
|
|
|
|
|
|
Randomly read 1 million files:
|
|
|
```
|
|
|
Concurrency Level: 16
|
|
|
-Time taken for tests: 34.263 seconds
|
|
|
+Time taken for tests: 22.301 seconds
|
|
|
Complete requests: 1048576
|
|
|
Failed requests: 0
|
|
|
-Total transferred: 1106762945 bytes
|
|
|
-Requests per second: 30603.34 [#/sec]
|
|
|
-Transfer rate: 31544.49 [Kbytes/sec]
|
|
|
+Total transferred: 1106812873 bytes
|
|
|
+Requests per second: 47019.38 [#/sec]
|
|
|
+Transfer rate: 48467.57 [Kbytes/sec]
|
|
|
|
|
|
Connection Times (ms)
|
|
|
min avg max std
|
|
|
-Total: 0.0 0.5 20.7 0.7
|
|
|
+Total: 0.0 0.3 54.1 0.2
|
|
|
|
|
|
Percentage of the requests served within a certain time (ms)
|
|
|
- 50% 0.4 ms
|
|
|
- 75% 0.5 ms
|
|
|
- 95% 0.6 ms
|
|
|
- 98% 0.8 ms
|
|
|
- 99% 1.2 ms
|
|
|
- 100% 20.7 ms
|
|
|
+ 50% 0.3 ms
|
|
|
+ 90% 0.4 ms
|
|
|
+ 98% 0.6 ms
|
|
|
+ 99% 0.7 ms
|
|
|
+ 100% 54.1 ms
|
|
|
```
|
|
|
|
|
|
+[Back to TOC](#table-of-contents)
|
|
|
|
|
|
-## License
|
|
|
+## License ##
|
|
|
|
|
|
Licensed under the Apache License, Version 2.0 (the "License");
|
|
|
you may not use this file except in compliance with the License.
|
|
|
@@ -450,7 +546,10 @@ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
|
|
See the License for the specific language governing permissions and
|
|
|
limitations under the License.
|
|
|
|
|
|
+The text of this page is available for modification and reuse under the terms of the Creative Commons Attribution-Sharealike 3.0 Unported License and the GNU Free Documentation License (unversioned, with no invariant sections, front-cover texts, or back-cover texts).
|
|
|
+
|
|
|
+[Back to TOC](#table-of-contents)
|
|
|
|
|
|
-## Stargazers over time
|
|
|
+## Stargazers over time ##
|
|
|
|
|
|
[](https://starcharts.herokuapp.com/chrislusf/seaweedfs)
|
yourchanges