Acquiring Create Date before downloading FTP file using iNet control

[Chrisikins]

I've searched the internet and this seems to be a common problem, but never satisfactorily resolved. I'm trying to get the create date and time of a file before downloading it.

The reason is that I need to download a file, the creation and name of which is beyond my control, but which has a name which includes the date, eg. SK080818.txt for 18th August O8. The problem is that the file is updated TWICE daily and has the same name for the morning version as for the different evening version.

It is about 46Mbytes, so I don't want to download it if already downloaded, and thus the create TIME is important.

The file is on an FTP site but I have been unable to persuade the Microsoft iNet control to give me the header data so that I can identify whether this is a new file or not. Obviously, the first of the day is easy because it has a different name, but the second of the day means that I need to check the create time first.

The iNet.getheader command only appears to work on HTTP files and not FTP files for some reason.

Does anyone have any suggestions, please?

Thank you in anticipation ...

Chris.

[jmsrickland]

I could be wrong but I think you will need to use FTP commands with the Inet Control.

I know there are examples in this forum that show you how to use some FTP commands and I think you will need to add some more like LIST which returns the directory information of all files or a single file.

LIST no-argument returns a directory listing of all files
LIST blablabla returns the directory listing for the file blablabla.

You will need to know the path where the file is and if it is not the root directory then you will need to use the CD path command to get to the correct directory

Search for things like FTP Download, FTP Upload, Inet.

[Chrisikins]

Thank you for your reply.

Yes, I am working in FTP mode.

Yes, it is in a sub-directory, and I have already programmed it to download a list of the contents of the sub-directory, but it only includes file names, not creation dates and that is the problem.

With the download of the subdirectory listing (names only), I can determine if a new file has been placed there that day, but cannot determine the difference between the morning and evening data files, which have identical names.

So far as I can see, there is no method of LIST with the iNet control, nor a parameter (e.g. icList) to pass to give the data/time stamp.

I have googled quite thoroughly, and have found others with similar, but unanswered questions.

So ... I am still baffled.

Thanks.

Chris.

edited to add: I've just re-read your post and am not sure what you mean by "FTP commands" - do you mean something outside of the iNet control? I should say that I have programmed quite a lot in VB before, but never previously done anything connected with the Internet. .... C.

[jmsrickland]

Yes, Inet allows you to send FTP commands. It's something like this:

Inet1.Execute URL, Operation, InputData

Where...

URL is the URL of the WebSite
Operation is any FTP command
InputData is any data needed to complete the operation (if required)

For a LIST command it would be like this:

Inet1.Execute URL, "LIST"

...there is no InputData for the LIST command

So in a typical Inet configuration you would do something like this:

Inet1.Protocol = icFTP
Inet1.RemoteHost = "www.some-web-site.com"
Inet1.UserName = "your-user-name"
Inet1.Password = "your-password"
Inet1.Execute , "CWD directory-path"
Inet1.Execute , "LIST filename"
etc
etc

Now I'm not sure if above is 100% absolutely correct but I think it is something like that.

Also, I am not sure of the exact format of the LIST command above but I know there is a command or a variation of the LIST command that will retrieve the file info including the timestamp.

[Chrisikins]

Thank you very much for that, BUT I can't find any reference to the LIST command. Not sure if I'm being thick or blind or what, but the only ones I can find are (Remember that we are in the VB6 version):

inet1.execute ,"PUT localfilename remotefilename" ' this is for uploading
inet1.execute ,"GET remotefilename localfilename" ' this is for downloading
inet1.execute ,"DIR" 'for the directory - LIST OF FILENAMES ONLY
inet1.execute ,"CD dirname" ' for changing DIrectory
inet1.execute ,"MKDIR newdir" ' for creating Directory
inet1.execute ,"RMDIR dirname" ' for removing the directory
inet1.execute ,"RENAME filename newfilename" ' for renaming the files
inet1.execute ,"DELETE filename" ' for deleting files
inet1.execute ,"CDUP" ' for levelup
inet1.execute ,"CLOSE" ' for closing the connection

It doesn't like LIST at all. The error message for inet1.execute ,"LIST " & [filename] or [urlpath & filename] is:

"Run-time error '35753':
Protocol not supported for this method."

Additionally, the "inet1.getheader" command gives the error message,

"Run-time error '35765':
This call is not valid for an FTP connection."

I'm afraid that I still can't see a method by which one can obtain the header information pre-download. It can, of course, be done post-download; however, as soon as I download, the timestamp naturally changes to the date/time at which it was downloaded, so post-download data is as worthless as not being able to access the timestamp pre-download.

I apologise for being completely dim, but I seem to be having a blindspot here somewhere and I'm not sure where it is.

Heeeeeeeeeeeeelp!

Thank you.

Chris.

Oh, why didn't I chose another language. Why, oh why can't we easily drop into machine code and write our own drivers like we used to do for mainframes 30 years ago? You can tell an ex-Coral programmer!

[Siddharth Rout]

Does this help?

http://msdn.microsoft.com/en-us/library/aa227495.aspx

Check for LoadRemoteInfo Routine and then how to analyse the information...

Hope this helps...

[Chrisikins]

An initial glance and it is looking very promising - thank you very much indeed.

I know from other questions around the internet on this topic that I am not alone with this question.

If this resolves it, I will drop the additional code in here as an example for the benefit of others.

Now I'll go away and study in ... much more fun than going to bed.

Chris.

[jmsrickland]

You could always use Winsock to do your FTP with. It's a bit more involved but you get all the FTP commands and more. If interested let me know.

[jmsrickland]

OK, now I see why you can't use LIST. It appears that the Inet control only supports high level commands (not actual FTP commands) which are macro like commands that internally do what the actual FTP commands would do. So, LIST is an actual FTP command and I guess there is no high-level equivalent of this command unles it's the short version of LIST and it becomes DIR when used with Inet.

Example:

Inet uses CD but the actual FTP command is CWD and GET and PUT are actually RETR and STOR in FTP lingo. In otherwords, there is no FTP command CD, GET, PUT, etc.

Using Winsock on the other hand you have to use the actual FTP commands.

[Chrisikins]

Yes, that would be appreciated, thank you.

I'm looking at Winsock and from what I can gather, it appears that one would need to CONNECT then get the first chunk of data using GETDATA, and parse the header data from that first chunk ... or is there an easier way?

I should say that the data file is placed in a sub directory, if that makes any difference in this case, although the subdirectory name is fixed permanaently.

At the same time as showing me how to do it, may I also ask the following:

When connecting using WINSOCK, can one cheat using the ftp://[username]:[password]@[domain]/[subdirectory], or does one have to supply the domain, username, password and subdirectory separately as one has to in the iNet case? (actually, one can use the single line construct in iNet, so presumably one can in winsock?)

Additionally, if one uses Winsock for the download, I presume it is possible to use the LOF data to calculate percentage downloaded/remaining, and thus the likely time remaining, etc., which would be an advantage, but to do that after each chunk downloaded is trivial, so long as I have the LOF.

Thanks again and I look forward to your advice.

Chris.

[jmsrickland]

Chris,

First thing first. I haven't tried using the single line with Winsock but since it's Winsock one could definitly send that string to the FTP server but I dwouldn't know what you would get back in return other than a HTTP type page showing the directory and it's contents. Try using it with IE and what you get back with IE would be the same as what you would get back using Winsock.

I'm looking at Winsock and from what I can gather, it appears that one would need to CONNECT then get the first chunk of data using GETDATA, and parse the header data from that first chunk ... or is there an easier way?

No. That is not how it works.

I won't go into alot of detail at this time but here is the general way you do it.

First, you need two Winsock controls on your VB Form. Why? Because FTP is a duo-channel communication session.

One Winsock control is used to send commands only. The other Winsock control is used to send and retrieve data. So, you have a command channel and a data channel going on at the same time.

Using the command channel (let's say your Winsock control is called sckCommand) you do the following:

sckCommand.Connect "www.somewebsite.com", 21

Note that you do not include ftp://. The ,21 is the port number and port 21 specifies FTP (as opposed to port 80 for HTTP).

After you connect the server will return a greeting message in your sckCommand_DataArrival Sub. It could look something like this:

220 plus1.host4u.net FTP server ready. <--- This is my FTP server

Once you recieve the 220 message you know you are connected and the server is now waiting for you to sign-in.

So, you respond with

sckCommand.Send "USER username" & vbCrLF

If the username is correct the server responds with this:

331 Password required for username

You respond with this:

sckCommand.Send "PASS your-password" & vbCrLf

If the password is correct the server will respond with this:

230 User username logged in

At this point you are now ready to start sending FTP commands to the server.

One of the first FTP commands you might want to send is this:

sckCommand.Send "TYPE A " & vbCrLf

This tells the server you want to deal with ASCII files (like a directory listing for example). Later when you want to deal with uploading or downloading files you set the TYPE to the type of file; that is, if you want to download a binary file you would send TYPE I but that stuff is for later.

The server will respond to the above with this:

200 Type set to I.

Maybe next you might want to see what directory you are in so you can do this:

sckCommand.Send "PWD " & vbCrLf

Which means Print Working Directory and the server would respond with this:

257 "/home/some-directory" is current directory.

Let's say that this is not the directory you want so you will need to tell the server to change to another directory. So you would send something like this:

sckCommand.Send "CWD the-new-directory-path " & vbCrLf

The server would respond with this:

250 CWD command successful.

OK, this is just a very general session with a FTP server. We haven't even got to the data channel at this time. That comes later when and if you want to send or retrieve data.

Look at what I have posted above and let me know if you want to get into FTP using Winsock. Like I said earlier it's a bit more involved (more detail than using Inet) but that's the way it works using Winsock in a FTP session.

[Chrisikins]

Thank you for that.

Yes, it looks as though I don't have any choice, even though I was hoping to avoid the Winsock route. Still, it's all a learning experience.

As always, official documentation expects one to be psychic and have more knowledge than most of us have in reality, which is why your tutorial is so helpful.

I will now put into place the sequence you have shown. I can't see any reference elsewhere about putting two copies of the control on the form.

I will follow your instructions above and see where we go from there. I can do the actual downloads, etc., in iNet, but Winsock will definitely be needed to determine the timestamp.

Chris.

[jmsrickland]

I am working on a simple FTP program for you and I will post the code when I get done which will be in just a few minutes.

[Chrisikins]

Thank you. That is very generous of you. I will hold off, then until I see it.

As I said, the rest of it works and I've done the asynchronous download part; I only need to establish the creation time of the file in order to determine whether it needs to be downloaded or not.

Chris.

[AsmIscool]

LIST is only used on it's own line by my FTP client. There is no filename and it gives a full directory listing with date and time. You could then iterate through that.

[jmsrickland]

OK, here it is.

On a Form put a Command button, a Textbox, and two Winsock controls.

To make the reading look good change the Font of the Textbox to Fixedsys.

Name one Winsock control sckCommand and the other sckData.

Copy and Paste the below code into the VB project and make the necessary changes.

Private DataPort As Integer
Private CurrentCommand As String
Private Processing As Boolean
Private ReturnCode As String
Private ReturnInfo As String
Private Connected As Boolean

Private Sub Command1_Click()
 Connected = False
 sckCommand.Close
 sckCommand.Connect "www.your-website.com", 21
End Sub

Private Sub Form_Load()
 DataPort = 7779 - 10
End Sub

Private Sub sckCommand_Connect()
 Do While Connected = False
   DoEvents
 Loop
 Text1.Text = Text1.Text & "Server:  " & ReturnCode & " " & ReturnInfo & vbCrLf
 '-----------------------------------------------------------------------------------------------------------------
 '
 ' Send your uersname to the server
 '
 Text1.Text = Text1.Text & "Client:  USER your-user-name" & vbCrLf
 CurrentCommand = "USER"
 Processing = True
 sckCommand.SendData "USER your-user-name" & vbCrLf
 Do While Processing
   DoEvents
 Loop
 Text1.Text = Text1.Text & "Server:  " & ReturnCode & " " & ReturnInfo & vbCrLf
 '-----------------------------------------------------------------------------------------------------------------
 '
 ' Send your password to the server
 '
 Text1.Text = Text1.Text & "Client:  PASS your-password" & vbCrLf
 CurrentCommand = "PASS"
 Processing = True
 sckCommand.SendData "PASS your-password" & vbCrLf
 Do While Processing
   DoEvents
 Loop
 Text1.Text = Text1.Text & "Server:  " & ReturnCode & " " & ReturnInfo & vbCrLf
 '-----------------------------------------------------------------------------------------------------------------
 '
 ' Switch to ASCII mode
 '
 Text1.Text = Text1.Text & "Client:  TYPE A" & vbCrLf
 CurrentCommand = "TYPE"
 Processing = True
 sckCommand.SendData "TYPE A" & vbCrLf
 Do While Processing
   DoEvents
 Loop
 Text1.Text = Text1.Text & "Server:  " & ReturnCode & " " & ReturnInfo & vbCrLf
 '-----------------------------------------------------------------------------------------------------------------
 '
 ' Print name (path) of the current working directory
 '
 Text1.Text = Text1.Text & "Client:  PWD" & vbCrLf
 CurrentCommand = "PWD"
 Processing = True
 sckCommand.SendData "PWD" & vbCrLf
 Do While Processing
   DoEvents
 Loop
 Text1.Text = Text1.Text & "Server:  " & ReturnCode & " " & ReturnInfo & vbCrLf
 '-----------------------------------------------------------------------------------------------------------------
 '
 ' Change to a given directory (CWD = Change Working Directory)
 '
 ' Commented out because I'll leave this up to you
 '
 'Text1.Text = Text1.Text & "Client:  CWD" & vbCrLf
 'CurrentCommand = "CWD"
 'Processing = True
 'sckCommand.SendData "CWD path-to-the-directory" & vbCrLf
 'Do While Processing
 '  DoEvents
 'Loop
 'Text1.Text = Text1.Text & "Server:  " & ReturnCode & " " & ReturnInfo & vbCrLf
 '-----------------------------------------------------------------------------------------------------------------
  
 '
 ' Below will get you the timestamp for a given file in the current directory or the directory
 ' you switched to if you used the above CWD command
 '
 CommandMDTM
End Sub

Private Sub sckCommand_DataArrival(ByVal bytesTotal As Long)
 Dim s As String
 Connected = True

 sckCommand.GetData s
 
 ReturnCode = Left(s, 3)
 ReturnInfo = Mid(s, 5)
 Processing = False
End Sub

Private Sub CommandMDTM()
 '
 ' This Sub is used so you can extract the timestamp from any given file in the current working directory
 '
 Dim FileName As String

 Dim FileYear As String
 Dim FileMonth As String
 Dim FileDay As String
 Dim FileHour As String
 Dim FileMinutes As String
 Dim FileSeconds As String
 
 Dim AMPM As String
 
 FileName = "file-name"
 
 Text1.Text = Text1.Text & "Client:  MDTM " & FileName & vbCrLf
 
 CurrentCommand = "MDTM"
 Processing = True
 
 sckCommand.SendData "MDTM " & FileName & vbCrLf
 
 Do While Processing
   DoEvents
 Loop
 Text1.Text = Text1.Text & "Server:  " & ReturnCode & " " & ReturnInfo & vbCrLf
 '-----------------------------------------------------------------------------------------------------------------
 FileYear = Left(ReturnInfo, 4)
 FileMonth = Mid(ReturnInfo, 5, 2)
 FileDay = Mid(ReturnInfo, 7, 2)
 FileHour = Mid(ReturnInfo, 9, 2)
 FileMinutes = Mid(ReturnInfo, 11, 2)
 FileSeconds = Mid(ReturnInfo, 13, 2)
 
 If Val(FileHour) < 12 Then AMPM = "AM"
 If Val(FileHour) = 12 Then AMPM = "Noon"
 If Val(FileHour) > 12 Then AMPM = "PM": FileHour = FileHour - 12
 If Val(FileHour) = 0 Then AMPM = "Midnight": FileHour = "12"
 
 Text1.Text = Text1.Text & "Client:  " & FileName & " Timestamp = " & FileMonth & "/" & FileDay & "/" & FileYear & " " & FileHour & ":" & FileMinutes & ":" & FileSeconds & " " & AMPM
End Sub

Private Sub CommandLIST()
 '
 ' This Sub will get you a full directory listing of all folders and files
 ' in the current working directory (Will not return timestamp - See CommandMDTM for that)
 '
 CommandPORT
 
 Text1.Text = Text1.Text & "Client:  LIST" & vbCrLf & vbCrLf & vbCrLf
 CurrentCommand = "LIST"
 Processing = True
 sckCommand.SendData "LIST -a" & vbCrLf
 Do While Processing
   DoEvents
 Loop
 Text1.Text = Text1.Text & "Server:  " & ReturnCode & " " & ReturnInfo & vbCrLf
End Sub

Private Sub CommandPORT()
 '
 ' The PORT command is necessary anytime you request data to send or recieve
 ' and must be issued prior to the data command (Like LIST, RETR, STOR)
 '
 Dim IPAddr As String
 
 CurrentCommand = "PORT"
 
 DataPort = DataPort + 10
 
 IPAddr = Replace(sckData.LocalIP, ".", ",") & ","
 IPAddr = IPAddr & DataPort \ 256 & "," & (DataPort Mod 256)

 On Error Resume Next
 
 sckData.Close
 sckData.LocalPort = DataPort
 sckData.Listen

 Text1.Text = Text1.Text & "PORT " & IPAddr & vbCrLf
 CurrentCommand = "PORT"
 Processing = True
 sckCommand.SendData "PORT " & IPAddr & vbCrLf
 Do While Processing
   DoEvents
 Loop
 Text1.Text = Text1.Text & "Server:  " & ReturnCode & " " & ReturnInfo & vbCrLf
End Sub

Private Sub sckData_ConnectionRequest(ByVal requestID As Long)
 sckData.Close
 sckData.Accept requestID
End Sub

Private Sub sckData_Connect()
 '
 ' Will never be entered because client never connects back to the server. Once the server connected to the Data socket
 ' then that socket takes over
 '
End Sub

Private Sub sckData_DataArrival(ByVal bytesTotal As Long)
 Dim s As String
 
 If bytesTotal = 0 Then Exit Sub
  
 sckData.GetData s, vbString
 
 Text1.Text = Text1.Text & s & vbCrLf
End Sub

Private Sub sckData_Close()
 '
 ' Enters here when the server has finished sending all the data
 '
 sckData.Close
 Text1.Text = Text1.Text & "Client:  Data Transfer Completed" & vbCrLf
End Sub

[jmsrickland]

LIST is only used on it's own line by my FTP client. There is no filename and it gives a full directory listing. You could then iterate through that.

Wrong.

You can do LIST no argument for a full directory listing

or

you can do LIST filename for only one file

Also, LIST does not return the timestamp. You need to use another command for that. See my code post.

[Chrisikins]

Again, thank you very much indeed.

Four hours of playing with it and setting it up, interrupted by the Opera problem (see below) sees the entire process working fully now

I now see why you need two Winsock controls and it all makes sense. From that, and having "SPLIT" the list into array lines, I can now use the list result to determine which file, if any, should be downloaded.

From the knowledge gained, I've also been able to add a button to test the domain, subdirectory, userID and password fields.

I did have one problem in that the entire process appeared to work down to the "Port command successful" statement, but INTERMITTENTLY, the list data didn't appear. However, this was resolved by closing Opera (browser), and further trials show that there is interference from Opera sometimes but not consistently; however, without a lot of research (which is not worth doing as I shall be using a dedicated machine), I don't know why that is happening.

Finally, because of the Opera problem and because my internet line has the stability of a yo-yo, I also included a method of checking for timeout on the waits for a response. If anyone would like it, let me know and I will happily post it.

Finally, if you will send your home or office postal address, I will forward an appropriate bottle as thanks for your your time and effort in helping me on this.

Chris.

[jmsrickland]

Don't you think it is better to use the MDTM command instead of the LIST to get your timestamp? Using the LIST requires the data socket, and the PORT command but the MDTM is not a data command so it is executable without having to set up your data channel and the return info is more straight forward.

[jmsrickland]

OK, I want to make a slight correction on post #17

Getting the timestamp.

It turns out that some FTP servers will return the timestamp using the LIST command. I tested my FTP code on a single FTP site and that site did not return the timestamp with the LIST command and that is why I said that LIST does not return the timestamp.

So, I tested that same code on another FTP site and that site did return the timestamp with the LIST command but only for the current year. So here is what it shows depending on when the file was first created or last modified.

-rw-r--r--    1 2656     513       5145598 Sep 19  2005 BHOSamples.zip
-rw-r--r--    1 2656     513       2707456 Jul 17 12:29 CompaqDrivers.zip

On the first line above the file was created Sept 19, 2005 and thus no timestamp is returnrd (by timestamp I mean HH:MM).

On the second line the file was created July 17, 2008 and a timestamp was returned as 12:29.

I looked at all entries in the list and noticed that all files and directories created prior to the current year only returned the year it was created and all files and directories created in the current year returned the HH:MM timestamp.

So, the conclusion is that one FTP site did not return the timestamp for any file/directory no matter when it was created and the other FTP site returned the timestamp only if the file/directory was created in the current year.

On both FTP sites using the MDTM command returned a timestamp for all files and directories in the format of YYYYMMDDHHMMSS.

So it turns out that you can't always depend on the LIST command to return a timestamp but you can always depend on the MDTM command to do so.

[Chrisikins]

Thank you for that point.

I've already written the parser for the LIST output, and it works. I will change it to take account of the special case you mention, although none of the download files are ever more than a week old.

Actually, there's a problem: I've tried using the MDTM command instead of the List, but all it ever returns is the the previous data (e.g. CWD response). I've tried putting the filename in as an absolute, but whatever I do, it doesn't seem to return anything. Also tried it with .GETDATA, and with a full path, but I'm obviously doing something wrong, but can't work out what.

In any case, MDTM, which is designed to return the datestamp for a single file should only return the data for ONE file, which is already available for all the files in the folder from the LIST command. It would be tidy to get it that way with the file list from the LIST command, but I don't think it is vital in this case.

Although I don't need it, it would be nice to know what I'm doing wrong with the MDTM command!

Chris.

[jmsrickland]

There's probably a timing problem going on betewwen your program and the server. The MDTM returns exactly what I illustrated in post #20. There is no way it can return the CWD responce. If that is what you are getting then you are out of sync somewhere or you are not parsing out the server replies correctly. Remember, you should always wait until the server has completed each sequence of events before you can actually depend on the responce. That is why I put a Loop after each command otherwise if you don't wait you can and probably will get a responce out of order.

[Chrisikins]

Yes, I thought that. Eventually, I asked the company, who asked their software people, and the reply came back, "It is an old system and MDTM is not supported". Apparently, because it works and everyone else downloads manually, they don't see any need to upgrade from "about six years" ago.

It must (by today's standards) be very old in that case! At least we know what we are dealilng with. The response is simply a repeat of the previous response, whatever that happened to be.

Looks like I have to continue to get the (relatively limited) data via the LIST command from 'ye olde server'; however, I think that is adequate for my purposes, and is just a matter of parsing (already written) and working with what we have.

Chris.

[jmsrickland]

I have dealt with many FTP servers over the years and never heard of that one.

Usually if a command is not supported the server will say so in a responce like

500 Unknown command

or

500 'MDTM': command not understood.

or

500 Command not supported.

but to simply return the previous responce is really wierd.

Anyway you are very fortunanate that the timestamp is part of the LIST return because as I mentioned above that not all servers include the timestamp with the LIST.

[Chrisikins]

Thank you again.

Well, of course, something else may well be going on, but that is what appears to be happening.

I will send you the details by private message and you will see what I mean; perhaps there is something else unusual about this server!

The files are added a couple of times a day and SOMETIMES they remove the old files, but not always.

I have programmed to find the latest file and to download if there is a new file, obviously storing the time data from the previous download, but to use MDTM to get the timestamp would be a nice bonus.