Upload file as chuncks

[danyamasadeh]

Hi
I am working on a XOJO project that I have to split large file into small chunks.

I read the code here:
http://www.boredomsoft.org/file-uploads-form-encodings-and-xojo.bs

It is working great on uploading a single file, but when trying to split the file into chunks and upload them in loop (so each chunk as a file) it is not working but sometimes it uploads one chunk and the others usually I got upload file error = UPLOAD_ERR_PARTIAL

Is there something I have to add to the content or header?

Thanks for your help

[charonn0]

I'm not sure I understand what you mean. Are you trying to upload a large file as a single form element without loading the whole thing into memory, or is the server expecting a form with a single file split into several elements?

[danyamasadeh]

I am trying to split the file in XOJO, something like:
Dim bs As BinaryStream = BinaryStream.Open(File)
bs.Position = offset
out.Write(bs.Read(NumOfBytes) + CRLF)
bs.Close
then upload this chunk as a separate file, I am sending the chunk number and number of chunks in the form data, then on server side when I have all chunks I assemble them in one file.

[charonn0]

I'm not sure why you'd want to, but this can be done by adding a final ElseIf clause to the SetFormData method:

Sub SetFormData(sock As HTTPSocket, FormData As Dictionary, Boundary As String)
  If Boundary.Trim = "" Then
    Boundary = "--" + Right(EncodeHex(MD5(Str(Microseconds))), 24) + "-bOuNdArY"
  End If
  
  Static CRLF As String = EndOfLine.Windows
  Dim data As New MemoryBlock(0)
  Dim out As New BinaryStream(data)
  
  For Each key As String In FormData.Keys
    out.Write("--" + Boundary + CRLF)
    If VarType(FormData.Value(Key)) = Variant.TypeString Then
      out.Write("Content-Disposition: form-data; name=""" + key + """" + CRLF + CRLF)
      out.Write(FormData.Value(key) + CRLF)
    ElseIf FormData.Value(Key) IsA FolderItem Then
      Dim file As FolderItem = FormData.Value(key)
      out.Write("Content-Disposition: form-data; name=""" + key + """; filename=""" + File.Name + """" + CRLF)
      out.Write("Content-Type: application/octet-stream" + CRLF + CRLF) ' replace with actual MIME Type
      Dim bs As BinaryStream = BinaryStream.Open(File)
      out.Write(bs.Read(bs.Length) + CRLF)
      bs.Close
      
      ' BEGIN ADDED PART
    ElseIf FormData.Value(Key) IsA Dictionary Then
      Dim d As Dictionary = FormData.Value(Key)
      Dim bs As BinaryStream = d.Value("stream")
      Dim fn As String = d.Value("filename")
      Dim offset As UInt64 = d.Value("offset")
      Dim length As UInt64 = d.Value("length")
      out.Write("Content-Disposition: form-data; name=""" + key + """; filename=""" + fn + """" + CRLF)
      out.Write("Content-Type: application/octet-stream" + CRLF + CRLF) ' replace with actual MIME Type
      bs.Position = offset
      out.Write(bs.Read(length) + CRLF)
      ' END ADDED PART
    End If
  Next
  out.Write("--" + Boundary + "--" + CRLF)
  out.Close
  #If RBVersion > 2012 Then
    sock.SetRequestContent(data, "multipart/form-data; boundary=" + Boundary)
  #else
    sock.SetPostContent(data, "multipart/form-data; boundary=" + Boundary)
  #endif
End Sub

And then:

  Dim form As New Dictionary
  Dim file As FolderItem = GetOpenFolderItem("")
  Dim bs As BinaryStream = BinaryStream.Open(file)
  
  Dim part1 As New Dictionary
  part1.Value("filename") = file.Name + "1"
  part1.Value("stream") = bs
  part1.Value("offset") = 0
  part1.Value("length") = 512
  form.Value("part1") = part1
  
  Dim part2 As New Dictionary
  part2.Value("filename") = file.Name + "2"
  part2.Value("stream") = bs
  part2.Value("offset") = 512
  part2.Value("length") = 512
  form.Value("part2") = part2
  
  Dim h As New HTTPSocket
  SetFormData(h, form, "")
  bs.Close
  
  Dim s As String = h.Post("http://www.example.com/", 10)

[danyamasadeh]

Thanks for your help

What I want simply is to send multi requests same time (split big file to chunks and send each 10 files at the same time).
Using same socket or multi sockets or even threads always I could upload 13 files for example from 240 and the others I got "UPLOAD_ERR_PARTIAL" or even didn't receive from server.
The server settings are fine regarding max upload or size.

[charonn0]

I not sure that can be done with the HTTPSocket class. Maybe if you explained why you need to split up the file in the first place I could give a better suggestion.

[danyamasadeh]

The task is:
When any file has been added to a specific folder selected by the user it should be uploaded to the server.
Your example works great with small files but what if the user add a 1 GB file. so I thought the best is to split it to chunks and upload each 10 or 15 sametime with Keep_alive connection.

Hope it is clear.
Appreciate your help.

[charonn0]

The HTTPSocket doesn't support keep-alive; it closes the socket after the first request. You may want to look into using curl or libcurl.

[danyamasadeh]

Thanks.
I will try it.