Wednesday, June 13, 2007

Web Analysis with VFP: Download, Parse, and Stage

My Web Analysis application is beginning to take shape. At the moment, I have created a little road map document (which I'll share shortly), defined all my base classes (the entire application will be made available in the coming months), and proved the general concept (see below).

To recap, this project is about building a tool to help me make better business decisions on and using the Internet. While there are a ton of canned products on the market, I feel that I can get what I need with VFP (and get it better). In general, I want to be able to download a variety of data from my server, analyze it, and use that data to make decisions (everything from what IPs to ban to how I might redesign a certain page). See my post "Business Intelligence Through Web Analysis" for more details.

The following zip file contains all the code necessary to prove the concept. It (a) establishes a database, (b) downloads the log files from the FTP server, and (c) parses the file, storing it in a stage directory, which I can then use to create a Dimensional Model for analysis purposes. Here's the programs:

In the zip, you will find:

  • wa_main.prg (the setup program, check the path and run this once)

  • functions.prg (a place to store my functions, this will become a class soon)

  • download.prg (set your server information up and run, it calls load_logfile(), which lives in functions.prg)

  • ftp/ftp.prg (FTP services class written by Robert Abram many moons ago -- still works great, requires wininet.dll)

Now that the concept has been proved, and I can visually analyze my log files, I will start to build an application in VFP9 around it. Stay tuned!

No comments: