Charset conversion of very large files - 2GB and more

Sample for ScriptUtils.ByteArray.CharSetConvert | Changes | Purchase | Download

Examples

     This sample contains a block-by-block character set conversion algorithm for a very large files, you can use it for files with any size, this code work also with files over 2GB or 4GB. It takes maximum of 5*BlockSize of memory to convert the data (5MB of memory for the 1MB block).
     The SourceCharSet may be only a charset with a constant char size, one, two or four bytes (and the BlockSize must be an integral multiple of the character size). You can use single byte charsets as Windows-1250, windows-1251, Latin II, ibm850... for the source file, but not utf-8 (it has a variable size of char). You can use any destination charset, including utf-8

Charset conversion of very large files - 2GB and more 
Option Explicit
FileConvert "H:\giga.dat", "utf-8", "Z:\giga-01.dat", "windows-1250" 
'FileConvert "H:\giga.dat", "windows-1250", "Z:\giga-01.dat", "windows-1250" 

'Charset conversion of a large file - 2GB, 4GB or more
'ScriptUtilities.ByteArray VBA sample
Sub FileConvert(SourceFileName, SourceCharSet, DestFileName, DestinationCharSet)
  'Set block size. Bigger block: better speed, more resources.
  Const BlockSize = &H100000
  
  Dim sPos, dPos, sByteArray, dByteArray, TotalBytes , ReadBlockSize
  
  Set sByteArray = CreateObject("ScriptUtils.ByteArray") 
   
  sByteArray.CharSet = SourceCharSet
  
  'Get a source file length.
  TotalBytes = GetFileSize(SourceFileName)
  
  'Clear the destination file
  DeleteFile DestFileName

'  If TotalBytes > 0 Then 'Convert only nonzero files.
    'For each data block from the source file
    For sPos = 0 To TotalBytes Step BlockSize
      Wscript.Echo sPos
      If BlockSize + sPos > TotalBytes Then
        'Last block - read only rest of data.
        ReadBlockSize = TotalBytes - sPos
      Else
        ReadBlockSize = BlockSize
      End If
      
      'Read the block of source data.
      sByteArray.ReadFrom SourceFileName, sPos + 1, ReadBlockSize
      
      'Convert the data to the destination charset
      Set dByteArray = sByteArray.CharSetConvert(DestinationCharSet)
      
      ' and store them to a disk. 0 means store the data at the end of file
      dByteArray.SaveAs DestFileName, 0
      
      'Increment the destination position
      dPos = dPos + dByteArray.Length
    Next ' sPos
'  End If ' TotalBytes > 0 Then
End Sub

'Get file size - 64 bit version, unlimited size
Function GetFileSize(FileName)
  Dim Kernel: Set Kernel = CreateObject("ScriptUtils.Kernel")
  On Error Resume Next
  GetFileSize = Kernel.GetFileSize(FileName)
  If err<>0 Then GetFileSize = 0
  On Error Goto 0
End Function

Function DeleteFile(FileName)
  Dim Kernel
  Set Kernel = CreateObject("ScriptUtils.Kernel")
  On Error Resume Next
  Kernel.DeleteFile FileName
  On Error Goto 0
End Function

Other links for the Charset conversion of very large files - 2GB and more sample

ScriptUtils.ByteArray

Works with safearray binary data - save/restore binary data from/to a disk, convert to a string/hexstring, codepage/charset conversions, Base64 conversion, etc.
     ByteArray is a COM class specially designed to work with Microsoft Windows Scripting engines - VB Script and JScript in Active Server Pages or WSH and in CHM or HTA applications. It also works with VB Net, Visual basic (VBA - VB 5, VB 6, Word, Excel, Access, ), C#, J#, C++, ASP, ASP.Net, Delphi and with T-SQL OLE functions - see Use ByteArray object article. You can also use the object in other programming environments with COM support, such is PowerBuilder.
     Source code for ByteArray is available within distribution license, please see License page for ASP file upload and ScriptUtilities.

ScriptUtils

Huge ASP upload is easy to use, hi-performance ASP file upload component with progress bar indicator. This component lets you upload multiple files with size up to 4GB to a disk or a database along with another form fields. Huge ASP file upload is a most featured upload component on a market with competitive price and a great performance . The software has also a free version of asp upload with progress, called Pure asp upload , written in plain VBS, without components (so you do not need to install anything on server). This installation package contains also ScriptUtilities library. Script Utilities lets you create hi-performance log files , works with binary data , you can download multiple files with zip/arj compression, work with INI files and much more with the ASP utility.

© 1996 - 2011 Antonin Foller, Motobit Software | About, Contacts | e-mail: info@pstruh.cz


Other Motobit links:   IISTracer, real-time IIS monitor   ASP file upload - upload files to ASP. 
ActiveX/VBSScript registry editor  ActiveX NT User account manager  Export MDB/DBF from ASP Active LogFile  Email export  ActiveX/ASP Scripting Dictionary object