The FTPdig Project

Sternberg Astronomical Institute, Moscow University

What is FTPdig?

• search through file/directory names, descriptions and README files;
• present directory subtrees in a graphical, easily understandable form;
• examine contents of archived (with tar/Zip/etc) files and view files from inside of such archives;
• download files and directory trees.

The need for this kind of interface to old-fashioned anonymous FTP service is great which is demonstrated by popularity of such services. Historically, probably the first widely known project of this kind was LEO (Link Everything Online); during last year, FTP search has gained a lot of attention. Several services of smaller scale are already operational or under development. However, both LEO and FTP search have their limitations: LEO is only serving its own ftp server (although very large one), FTP search is mostly oriented towards search with other functions being primitive or missing.

Goals and perspectives

The current goal of FTPdig project is to build University-wide network of ftp servers unified by the same interface and database. In other words, one can access several anonymous ftp servers as if it was single (and very large!) one, with all features mentioned in What is FTPdig available.

The project will be greatly helped by recent deploying of campus-wide fiberoptic link in Moscow University. This allows to distribute contents among many independent servers, interconnected by high-bandwidth link, just how it is currently done. Therefore no major reconstruction of anonymous ftp services is needed. The FTPdig interface and database can be mirrored on any local server, thus giving virtual instant access to all servers at once from any location. However, some services (like examining contents of archives) will still require access to original sites where actual files are kept.

In perspective, the project can be extended to cover the entire network of ftp servers in Russia. Architecture of existing implementation does not scale well above 1,000,000 files total. Expansion might require significant rewrite of underlying algorithms.

Technical background

The advantages of WWW interface are:

• archive user only needs WWW browser (and nowadays, almost every Internet user has WWW browser); when system is properly implemented, even users with outdated and primitive browsers can work with the archive;
• all processing (such as searching) is done on relatively powerful server machine;
• CGI scripts can provide sophisticated functions such as searching through entire database of file names/descriptions and displaying contents of compressed archives (*.tar.gz, *.zip, etc.), which are impossible to perform on user side without massive transfers.

• indexing software has to be installed on server; even simple software like File Juggler requires some amount of administrator work;
• workload of ftp server is significantly increased; while classic anonymous ftp service can be run on rather underpowered machine, CGI scripts with lots of processing need fast hardware (fast CPU and large amounts of random access memory);
• some ftp user agents have quite advanced capabilities, and these programs aren't constrained by HTML and CGI limitations.

The model of FTPdig distributed service has been

You might want to read also implementation notes on File Juggler, current incarnation of FTPdig project. Performance problems and limitation of current design are discussed there.

Current status and implementation

With rapid growth of SAI ftp archives, it has become a necessity to have a convenient search and navigation interface for them. Such interface was implemented by Dr. Sergey Ayukov, research fellow at Sternberg Astronomical Institute of Moscow University. The software was named "File Juggler"; it is considered the first stage of FTPdig Project.

File Juggler Version 1.0 is now installed on two ftp servers at Sternberg Astronomical Institute, Moscow University:

1. Crydee.sai.msu.ru archive (mostly OS/2 software, recreation and leisure)
2. SAI FTP archive (lots of free material available on the Internet, including astronomy, Unix software, Linux etc.)

Version 1.0 can only work with one ftp site at a time (on which it is installed). Next version which will provide interface to several ftp servers simultaneously is under development.

File Juggler uses cgic library, copyright 1996 by Thomas Boutell.