Written in Lua provides reliable uploads and easy to extend logic of file upload lifecycle. Currently only PUT/POST RAW body resumable uploads are supported. These are compatible with nginx-upload-module resumable requests. This extension requires Nginx compiled with HttpLuaModule.
v1.0.0 is deployed in production since 4 Apr 2013. ~50GB/day, ~100 users/day with mixed browsers.
For some form of documentation please see tests directory.
I created this module as an user of excellent nginx-upload-module which processed terabytes of data for several years. Sometimes I needed
more features which were missing such as CRC32 calculation for resumable uploads. Unfortunately I suspect that module may be abandoned, see #41,
therefore I started a development of new solution which will work with Nginx 1.3.8+ and will provide some area for enhancement. I prefer to use
Lua module which gives very short development/test iterations in reload per request mode. Here is Pros/Cons list of this module in comparison to nginx-upload-module:
- Pro: no status file because current offset is equal to size of linearly uploaded file;
- Pro: works with any version supported by Lua module including nginx 1.3.8+;
- Pro: this module supports the same resumable upload headers and response as
nginx-upload-module, thus it may be drop-in replacement in some deployments; - Pro: easy to enhance/fork, Lua modules are easy to develop, it took me 2 days to learn Lua and create this module;
- Cons: no multi-part POST format supported, only RAW file in PUT/POST request; check lua-resty-upload;
- Cons: slightly slower because of Lua layer, but this is very small factor IMO, check the benchmark below;
- Cons/Pro: linear resumable - there is no way to upload last chunk as first in row, chunks must be uploaded sequentially; this is Pro actually because prevents DoS attacks with malicious upload request;
- Cons: no upload rate limit setting - only one client can upload a given file so it is better to throttle downloads.
Below is the result of test/performance_test.rb method test_final_perf, which tests nginx-big-upload (bu) vs nginx-upload-module (num)
with chunked upload of identical data to the same backend. Each file is sent in ~512KB chunks which means 51MB file
is transferred in 100 requests. Times in seconds. Fileds user/system/total are rather related to test script execution times, but real is the real one.
user system total real
bu 51MB * 10 0.630000 0.820000 1.450000 ( 3.199759)
bu_crc 51MB * 10 0.670000 0.910000 1.580000 ( 4.164285)
num 51MB * 10 0.680000 0.920000 1.600000 ( 2.677721)
bu 204MB * 10 2.520000 3.510000 6.030000 ( 12.576817)
num 204MB * 10 2.480000 3.940000 6.420000 ( 10.818588)
- Nginx with Lua module built in. LuaJIT is preferred as runtime implementation. Hint: run
ldd nginxto make sure LuaJIT will be used. - To run tests you also need nginx with nginx-upload-module v2.2 and Ruby 2.0, but 1.9.x may also work.
Checksum of uploaded data can be provided by the client by X-Checksum request header. In case
of chunked upload that may be added in last chunk, because the header is passed to backend if Backend Handler is enabled.
Moreover Backend Handler puts checksum into backend request parameters.
To calculate checksum on server-side $bu_checksum on; configuration variable should be set. The server calculates
the checksum and Backend Handler includes it in the checksum param of backend request. If client provides 'X-Checksum'
then it is compared with server-side checksum. If they do not match a response code 400 is returned and further processing is stopped.
Server-side checksum with chunked upload requires client to pass partial checksums between chunks. This is because
server-side does not store any information to continue calculation of checksums for entire upload. The client
should remember the value of X-Checksum response header of last chunk and put it into X-Last-Checksum request header of next chunk.
Simplified example:
> PUT /upload
> Content-Range: bytes 0-4/10
>
> Part1
< 201 Created
< X-Checksum: 3053a846
> PUT /upload
> Content-Range: bytes 5-9/10
> X-Last-Checksum: 3053a846
>
> Part2
< 201 Created
< X-Checksum: 478ac3e5
Client can also provide own X-Checksum in upload request then checksums will be matched:
> PUT /upload
> Content-Range: bytes 0-4/10
> X-Checksum: 3053a846
>
> Part1
< 201 Created
< X-Checksum: 3053a846
> PUT /upload
> Content-Range: bytes 5-9/10
> X-Last-Checksum: 3053a846
> X-Checksum: 478ac3e5
>
> Part2
< 201 Created
< X-Checksum: 478ac3e5
- storage handler with back-end subrequest before first chunk - e.g. to check if upload is acceptable or to track temporary file
- upload status requests,
- back-end-side progress notifications,
- JavaScript example with HTML5 File API and file chunking with Blob.slice.
- cloud pass-thru upload handler,
- multiparts support - this is low priority please use excellent nginx-upload-module or lua-resty-upload