Performance tests of several methods to store files on server-side. Save file to disk, store to database.
'Code 1
Server.ScriptTimeout = 120
Dim Form: Set Form = Server.CreateObject("ScriptUtils.ASPForm")
If Form.State = 0 Then
Form.Files("MyFile").Save "f:\Uploads\UpFile"
End If
|
Field.Save | ||
|
|
Consumed | |
|
FileSize [MB] |
Memory [MB] |
Processor [ms] |
|
5 |
9.2 |
94 |
|
10 |
10.3 |
172 |
|
15 |
10.3 |
281 |
|
20 |
10.3 |
328 |
|
25 |
10.3 |
531 |
|
30 |
10.3 |
547 |
|
35 |
10.3 |
781 |
|
40 |
10.3 |
734 |
|
45 |
10.3 |
859 |
|
50 |
10.3 |
1 031 |
Save method consumes constant amount of memory (block size is set to 1MB). Processor time (save time) depends linearly on file length.
'Code 2
Function GetConnection()
Dim Conn: Set Conn = CreateObject("ADODB.Connection")
Conn.Provider = "Microsoft.Jet.OLEDB.4.0"
Conn.Open "Data Source=e:\upload2000.mdb"
Conn.CommandTimeout = 1800
Set GetConnection = Conn
End Function
'Code 3 'Open dynamic recordset, table Upload RS.Open "Upload", Conn, 2, 2 'Add a new record RS.AddNew 'update info fields in "Upload" table '.... 'Add file from source field 'DBFile' to table field 'Data' RS("Data") = Form("DBFile").ByteArray RS.Update
|
MDB, RS("Data") = ... | ||
|
|
Consumed | |
|
FileSize [MB] |
Memory [MB] |
Processor [ms] |
|
1 |
10.9 |
187 |
|
5 |
23.5 |
265 |
|
10 |
39.2 |
312 |
|
15 |
67.9 |
468 |
|
20 |
83.3 |
578 |
|
25 |
98.7 |
671 |
|
30 |
114.1 |
703 |
|
35 |
129.5 |
937 |
|
40 |
129.8 |
921 |
|
45 |
140.2 |
875 |
|
50 |
138.0 |
625 |
Memory consumption grows linearly, up to 140 MB, as consumed processor time. Then JetOLEDB provider stores data using blocks and some consumed time was moved to System process.
'Code 4
'Set block size to 64kB
Const BlockSize = &H10000
'Process source data using blocks
Dim BlockCounter, DataBlock
For BlockCounter = 0 To Form("DBFile").Length Step BlockSize
'Get a part of source data
DataBlock = Form("DBFile").ByteArray(BlockCounter, BlockSize)
'Add a part of source data to the field
RS("Data").AppendChunk DataBlock
Next 'BlockCounter
|
RS.AppendChunk - JetOLEDB | ||
|
|
Consumed | |
|
FileSize [MB] |
Memory [MB] |
Processor [ms] |
|
5 |
13.8 |
296 |
|
6 |
14.9 |
312 |
|
7 |
16.0 |
375 |
|
8 |
17.0 |
390 |
|
9 |
18.1 |
406 |
|
10 |
19.2 |
468 |
|
15 |
22.6 |
687 |
|
20 |
23.9 |
1 000 |
|
25 |
23.9 |
1 250 |
|
30 |
23.9 |
1 609 |
|
35 |
23.9 |
1 859 |
|
40 |
23.9 |
2 125 |
|
45 |
23.9 |
2 312 |
|
50 |
23.9 |
2 546 |
Memory consumption grows linearly, up to 20MB and then stops at 23.9MB. Consumed processor time grows also linearly.
'Code 5
'open connection for SQLOLEDB
Function GetConnection()
Dim Conn: Set Conn = CreateObject("ADODB.Connection")
Conn.Provider = "SQLOLEDB"
Conn.Open "Data Source=Muj;Database=Upload", "sa", ""
Set GetConnection = Conn
End Function
'open connection for MSDASQL
Function GetConnection()
Dim Conn: Set Conn = CreateObject("ADODB.Connection")
Conn.Open "DSN=Upload", "sa", ""
Set GetConnection = Conn
End Function
|
SQL 2000, RS("Data") = ... | ||
|
|
Consumed | |
|
FileSize [MB] |
Memory [MB] |
Processor [ms] |
|
1 |
8.1 |
234 |
|
5 |
26.6 |
343 |
|
10 |
47.1 |
234 |
|
15 |
67.6 |
315 |
|
20 |
88.1 |
406 |
|
25 |
108.7 |
515 |
|
30 |
129.2 |
616 |
|
35 |
149.7 |
722 |
|
40 |
170.3 |
856 |
|
45 |
190.6 |
984 |
|
50 |
211.4 |
1 111 |
I tested these two connections and get the same performance results. The results are in the table. The memory consumption grows linearly, but SQL provider takes 4Bytes to send one Byte to SQL! And MS SQL takes also some bytes.
|
RS.AppendChunk - SQLOLEDB | ||
|
|
Consumed | |
|
FileSize [MB] |
Memory [MB] |
Processor [ms] |
|
5 |
16.9 |
484 |
|
10 |
27.3 |
1 328 |
|
15 |
37.7 |
2 312 |
|
20 |
48.0 |
3 921 |
|
25 |
58.2 |
5 578 |
|
30 |
68.5 |
7 750 |
|
35 |
78.8 |
10 093 |
|
40 |
88.2 |
12 546 |
|
45 |
97.9 |
15 671 |
|
50 |
107.3 |
19 568 |
So let's go on it. I take the same code as with MDB (Code no. 4) and tested it with MS SQL 2000. And I must say - great idea, very bad implementation. As you can see, this code takes a little bit les memory than the code without chunks - only two bytes per one source file. But it has another big problem - consumed processor time depends by square on data length!
'Code 6
'Set block size to 1MB
Const BlockSize = &H100000
'Process source data using blocks
Dim BlockCounter, HexDataBlock
For BlockCounter = 0 To Form("DBFile").Length Step BlockSize
'Get a part of source data as a Hex string
HexDataBlock = Form("DBFile").HexString(BlockCounter, BlockSize)
'Add a part of source data to the field using SQL UPDATETEXT command
SQL = "DECLARE @dataptr binary(16)" & vbCrLf
SQL = SQL & "SELECT @dataptr = TEXTPTR(Data)"
SQL = SQL & " FROM Upload WHERE UploadID=" & UploadID & vbCrLf
SQL = SQL & "UPDATETEXT Upload.Data @dataptr " & BlockCounter & " NULL 0x" & HexDataBlock
'Execute prepared SQL command
Conn.Execute SQL, 0, adCmdText + adExecuteNoRecords
Next 'BlockCounter
|
HexString - SQLOLEDB | ||
|
|
Consumed | |
|
FileSize [MB] |
Memory [MB] |
Processor [ms] |
|
5 |
27.4 |
5 078 |
|
10 |
27.4 |
9 953 |
|
15 |
27.4 |
14 734 |
|
20 |
27.4 |
20 093 |
|
25 |
27.4 |
24 312 |
|
30 |
27.4 |
29 125 |
|
35 |
27.4 |
33 609 |
|
40 |
27.4 |
38 484 |
|
45 |
27.4 |
43 437 |
|
50 |
27.4 |
48 390 |
And how this code works?
For example 100MB file. Uploaded data are stored on the disk. We add another
fields of upload record to the database (some bytes/kB).
Then we process source binary data using
blocks. HexString reads 1MB of source data from the disk, converts this data to
HexString (+4MB - Unicode HexString), creates UPDATETEXT SQL (+4MB) and sends
this SQL command to MS SQL. MS SQL gets this string (+10MB ADO +2MB in SQL)
converts it to binary data (+1MB) and the data are stored to the database file.
Frequently asked questions about ScriptUtilities, Pure-ASP and Huge-ASP upload functionality.
Huge ASP upload is easy to use, hi-performance ASP file upload component with progress bar indicator. This component lets you upload multiple files with size up to 4GB to a disk or a database along with another form fields. Huge ASP file upload is a most featured upload component on a market with competitive price and a great performance . The software has also a free version of asp upload with progress, called Pure asp upload , written in plain VBS, without components (so you do not need to install anything on server). This installation package contains also ScriptUtilities library. Script Utilities lets you create hi-performance log files , works with binary data , you can download multiple files with zip/arj compression, work with INI files and much more with the ASP utility.
© 1996 - 2011 Antonin Foller, Motobit Software | About, Contacts | e-mail: info@pstruh.cz