To Encode String or to Not Encode String... Answer?

I am importing a CSV file from a Unix server as a string, and then saving the string into a new file for later processing.  However, this process does not always work.  As this CSV file is VERY, VERY large, I am unable to immediately tell if the fault lies with my code, or the CSV file provider (very possible).  The only time I can tell that this happened is when the data is not populated the next morning.  The fault appears to lie in there being extra or missing commas, as when I try to run the data import manually, STRINGs are attempted to be inserted into INTs and vice versa.

Should I explicitly be encoding the string in any specific way?  Could this be the culprit?

10/19/2007 2:37:42 PM 91979 articles. 4 followers. Follow

3 Replies

Similar Articles

[PageSpeed] 44
Get it on Google Play
Get it on Apple App Store

Can you import the file into Windows as one operation and read the CSV file in the next? If the fast reader may help and you can then parse each line separately.

As to quoting, you are best to quote each variable, irrespective of whether it needs it.

Don't forget to click "Mark as Answer" on the post that helped you.
This credits that member, earns you a point and marks your thread as Resolved so we will all know you have been helped.
10/19/2007 3:07:58 PM

The problem is not necessarily with the process, but rather with the end result.  I have no problem grabbing and saving the file locally (which I need to do).  However, the subsequent parsing is where the problem lies.  If I download and save this file manually, the parsing process runs without a hitch.  This is why I am thinking that this problem is related to the character encoding of the saved file during the automated process.

10/19/2007 5:24:16 PM


This is still unresolved and the problem persists.  Here is an example snippet of how I am grabbing the file:

1    Protected Function GetRemoteFile(ByVal RemoteFilePath As String, ByVal LocalFilePath As String) As String Implements IVendorDataImportBase.GetRemoteFile
3        Dim strResult As String = String.Empty
4        Dim resp As System.Net.HttpWebResponse = Nothing
6        Dim objUri As New Uri(RemoteFilePath)
7        If objUri.Scheme = Uri.UriSchemeHttp Then
8            ' begin grabbing the file
9            Dim req As System.Net.HttpWebRequest = CType(System.Net.HttpWebRequest.Create(objUri), System.Net.HttpWebRequest)
10           req.Method = System.Net.WebRequestMethods.Http.Get
11           resp = CType(req.GetResponse, System.Net.HttpWebResponse)
12           Dim rdr As New StreamReader(resp.GetResponseStream)
13           strResult = rdr.ReadToEnd
14           resp.Close()
15       End If
17       Return strResult
18   End Function
21   Protected Sub SaveLocalFile(ByVal FileName As String, ByVal FileText As String) Implements IVendorDataImportBase.SaveLocalFile
22       Dim fs As New System.IO.FileStream(FileName, FileMode.OpenOrCreate, FileAccess.Write, FileShare.None)
23       Dim fdata() As Byte = System.Text.Encoding.ASCII.GetBytes(FileText)
24       fs.Write(fdata, 0, fdata.Length)
25       fs.Flush()
26       fs.Close()
27   End Sub
11/5/2007 1:20:34 PM