freeswitch/libs/libetpan/doc/DOCUMENTATION

1/ Introduction
---------------

libEtPan! is mainly a library that will handle all kind of mailbox access.
For example: IMAPrev4, POP3, NNTP, mbox, MH.

You have two kinds of mailbox access, either using low-level functions
with a different interface for each kind of access or using higher-level
functions, using a driver to wrap the higher-level API.


2/ Low-level
------------

2.1/ IMAP4rev1 - Internet Message Access Protocol - Version 4rev1
-----------------------------------------------------------------

Each command of the IMAP4rev1 Standard (RFC 2060) is implemented in
the IMAP4rev1 module. Directory imap/.

2.1.1/ References

- RFC 2060 - INTERNET MESSAGE ACCESS PROTOCOL - VERSION 4rev1
- draft-crispin-imapv-15.txt

Not yet implemented :

- draft-crispin-imapv-20.txt

2.1.2/ Dependencies

- tools/

2.1.3/ Files descriptions

description of header files :
mailimap.[ch]              -- functions that implements each IMAP4rev1 command
mailimap_helper.[ch]       -- helper interface for the previous functions
mailimap_types.[ch]        -- definition of types and constructors for these
                                types
mailimap_types_helper.[ch] -- contains function definitions that will help
                              to create data necessary to use IMAP4rev1 module
mailimap_socket.[ch]       -- provides a function to connect to an
	                        IMAP4rev1 server over TCP
mailimap_ssl.[ch]          -- provides a function to connect to an
                                IMAP4rev1 server over TLS layer

2.1.4/ Interface

Include for this module is mailimap.h and includes all other headers.


The interface of IMAP4rev1 is documented in the following files :

mailimap.h
mailimap_types.h
mailimap_types_helper.h


2.2/ POP3 - Post Office Protocol - Version 3
--------------------------------------------

Each command of the POP3 Standard (RFC 1939 and RFC 2449) is implemented
in the POP3 module. Directory pop3/.

2.1.1/ References

- RFC 1939 - Post Office Protocol - Version 3
- RFC 2449 - POP3 Extension Mechanism (CAPA)

Not yet implemented :

- RFC 1734 - POP3 AUTHentication command

2.1.2/ Dependencies

- tools/

2.2.3/ Files descriptions

mailpop3.[ch]         -- functions that implements each POP3 command
mailpop3_helper.[ch]  -- helper interface for the previous functions
mailpop3_socket.[ch]  -- provides a function to connect to a
                           POP3 server over TCP
mailpop3_ssl.[ch]     -- provides a function to connect to a
                           POP3 server over TLS layer

2.2.4/ Interface

Include for this module is mailpop3.h and includes all other headers.

There is not yet documentation for POP3 module.


2.3/ NNTP - Network News Transfer Protocol
------------------------------------------

Each command of the NNTP Standard (RFC 977 and RFC 2980) is implemented
in the NNTP module. Directory nntp/.

2.3.1/ References

- RFC 977 - Network News Transfer Protocol
- RFC 2980 - Common NNTP Extensions

Not yet implemented :

- RFC 1036 - Standard for Interchange of USENET Messages
- son of RFC 1036 : FTP://zoo.toronto.edu/pub/news.txt.Z

2.3.2/ Dependencies

- tools/

2.3.3/ Files descriptions

newsnntp.[ch]         -- functions that implements each NNTP command
newsnntp_socket.[ch]  -- provides a function to connect to a
                           NNTP server over TCP
newsnntp_ssl.[ch]     -- provides a function to connect to a
                           POP3 server over TLS layer

2.3.4/ Interface

Include for this module is newsnntp.h and includes all other headers.

There is not yet documentation for NNTP module.


2.4/ mbox
---------

The mbox module provides a set of functions to manipulate mbox mailboxes.
These functions make a safe lock on the mailbox they work with.
This module will assign to each message a unique message identifier
so that we can work with message numbers in mbox files without other
programs interfer.
Directory mbox/.

2.4.1/ References

- http://wp.netscape.com/eng/mozilla/2.0/relnotes/demo/content-length.html
- http://www.qmail.org/qmail-manual-html/man5/mbox.html

2.4.2/ Dependencies

- tools/
- imf/

2.5.3/ Specific to libEtPan!

- "X-LibEtPan-UID" header

2.5.4/ Files descriptions

mailmbox.[ch]       -- functions to manipulate mbox mailboxes.
mailmbox_parse.[ch] -- this module is in charge of parsing the
                         mbox file content
mailmbox_types.[ch] -- definition of types and constructors for these
                         types

2.4.5/ Interface

Include for this module is mailmbox.h and includes all other headers.

There is not yet documentation for mbox module.


2.5/ MH
-------

The MH module provides a set of functions to manipulate MH mailboxes.
Directory mh/.

2.5.1/ References

- almost none

2.5.2/ Dependencies

- tools/

2.5.3/ Files descriptions

mailmh.[ch] -- functions to manipulate MH mailboxes.


2.5.4/ Interface

Include for this module is mailmh.h.

There is not yet documentation for MH module.


2.6/ IMF - Internet Message Format
----------------------------------

The IMF module provides functions to parse data given in RFC 2822
format (Internet Message Format).
Directory imf/.

2.6.1/ References

- RFC 2822 - Internet Message Format (Not entirely implemented)
- RFC 2076 - Common Internet Message Headers

Not yet implemented :

- RFC 2298 - An Extensible Message Format
               for Message Disposition Notifications

2.6.2/ Dependencies

- tools/

2.6.3/ Files descriptions

mailimf.[ch]              -- functions to parse RFC 2822 messages.
mailimf_types.[ch]        -- definition of types and constructors for these
                               types
mailimf_types_helper.[ch] -- contains function definitions that will help
                              to create data necessary to use IMF module.
mailimf_write.[ch]        -- functions that output RFC 2822 messages or
	                       sub-part of the messages in a (FILE *).

2.6.4/ Interface

Include for this module is mailimf.h and includes all other headers.

The interface of IMAP4rev1 is documented in the following files :

mailimf.h
mailimf_types.h
mailimf_types_helper.h
mailimf_write.h


2.7/ MIME - Multipurpose Internet Mail Extensions
-------------------------------------------------

The MIME module provides functions to parse structure of MIME messages.
Directory mime/.

2.7.1/ References

- RFC 2045 - Multipurpose Internet Mail Extensions (MIME) Part One: Format of
               Internet Message Bodies.
- RFC 2047 - MIME (Multipurpose Internet Mail Extensions) Part Three: Message
               Header Extensions for Non-ASCII Text. 
- RFC 2183 - Communicating Presentation Information in Internet Messages:
               The Content-Disposition Header Field

Not implemented :

- RFC 2046 - Multipurpose Internet Mail Extensions (MIME) Part Two: Media
               Types.

2.7.2/ Dependencies

- tools/
- imf/

2.7.3/ Files descriptions

mailmime.[ch]               -- functions to parse the MIME fields (RFC 2045).
mailmime_content.[ch]       -- functions to parse the MIME message. You get
                                 the different parts and you can decode them.
mailmime_decode.[ch]        -- functions to parse the MIME-encoded fields.
mailmime_disposition.[ch]   -- functions to parse the Content-Disposition field
                                 (RFC 2183)
mailmime_types.[ch]         -- definition of types and constructors for these
                                 types
mailmime_types_helper.[ch]  -- contains function definitions that will help
                                 to create data necessary to use MIME module.
mailmime_write.[ch]         -- functions that output MIME messages or
	                         sub-part of the messages in a (FILE *).

2.7.4/ Interface

Include for this module is mailmime.h and includes all other headers.

There is not yet documentation for MIME module.


2.8/ SMTP - Simple Mail Transfer Protocol
-----------------------------------------

Each command of the SMTP Standard (RFC 2821 and RFC 1891) is implemented
in the SMTP module. Directory smtp/.

2.8.1/ References

- RFC 2821 - Simple Mail Transfer Protocol (Not entirely implemented)
- RFC 1891 - SMTP Service Extension for Delivery Status Notifications

2.8.2/ Depencencies

- tools/

2.8.3/ Files descriptions

mailsmtp.[ch]         -- functions that implements each SMTP command
mailsmtp_helper.[ch]  -- functions to get an easier use of SMTP module
mailsmtp_socket.[ch]  -- provides a function to connect to a
                           SMTP server over TCP
mailsmtp_ssl.[ch]     -- provides a function to connect to a
                           SMTP server over TLS layer
mailsmtp_types.h      -- definition of types

2.8.4/ Interface

Include for this module is mailsmtp.h and includes all other headers.

There is not yet documentation for MIME module.


2.9/ Miscellaneous

2.9.1/ References

- RFC 2234 - Augmented BNF for Syntax Specifications: ABNF
- RFC 2595 - Using TLS with IMAP, POP3 and ACAP

2.9.2/ Tools

tools/ directory contains some tools functions and useful data structures.

alloc.h                 -- a wrapper on malloc()
carray.[ch]             -- an array, that grows automatically when elements
                             are added.
charconv.[ch]           -- character set converter. For example, it will
                             translate an iso-8859-1 string to an utf-8 string.
chash.[ch]              -- a hash table which keys can be anything
cinthash.[ch]           -- a hash table which keys are integers
                             (should be removed and replaced with chash)
clist.[ch]              -- a double-linked list
connect.[ch]            -- easy interface to connect a TCP server
hmac_md5.h
md5.[ch]
md5global.h             -- MD5 calculation
mail.h                  -- some constants
maildb_helper.[ch]      -- wrappers to DB 2.x 3.x or 4.x
maillock.[ch]           -- safely lock a given file
mailstream.[ch]         -- stream interface - buffered reading and writing
                             on files/socket/SSL connection
mailstream_helper.[ch]  -- useful functions for stream
                             (for example: read a line)
mailstream_low.[ch]     -- driver interface for a stream
mailstream_socket.[ch]  -- stream driver for file descriptors (includes socket)
mailstream_ssl.[ch]     -- stream driver for SSL connection
mailstream_types.h      -- data structure definition
mapping.[ch]            -- map parts of files in memory (no more used)
mmapstring.[ch]         -- a string, that grows automatically when data
                             are added.


3/ Higher-level
---------------

The higher level will allow us to query folder informations or to get
messages information or content.

There is four kinds of identities :
- storage
- folders
- session
- messages

In the higher-level interface, you manipulate data types from IMF and
MIME module, plus additionnal data types of higher-level.


3.1/ Objects
------------

3.1.1/ Storage

A storage (struct mail_storage) represents whether a server or
a main path, It can be an IMAP server, the root path of a MH or a mbox file.


3.1.2/ Folders

A folder can be created from a storage.
Folders (struct mail_folder) are the mailboxes we can choose in the
server or as sub-folder of the main path.

Folders for IMAP are the IMAP mailboxes, for MH this is one of the
folder of the MH storage, for mbox, there is only one folder, the
mbox file content;


3.1.3/ Session

Storage and folders communicate with the lower-layer through the
mail session data structure.

A mail session (struct mailsession) is a mail access to a server
or a mail access in the local file system. It allow us to send commands
to the mail access.

A mail storage is using a mail session to communicate.
A folder folder also uses a mail session to get information or to send
information. It can be the same session or not, depdending of the
implementation.


3.1.4/ Messages

From a session, we can get a message (struct mailmessage) to read.


3.2/ Drivers
------------

For a mail access, three drivers exist.
One for storage, one for session, one for message.
Note that the folder access rely only on session driver.


3.2.1/ storage driver interface

  mail_storage_driver is the driver structure for mail storage

  - name is the name of the driver
  
  - connect() connects the storage to the remote access or to
      the path in the local filesystem.
      
  - get_folder() can have two kinds of behaviour.
      Either it creates a new session and independant from the session
      used by the storage and select the given mailbox or
      it selects the given mailbox in the current session.
      It depends on the efficiency of the mail driver.

  - free_data() frees the data created with mail_storage constructor.

  a constructor for each kind of access has to be implemented.
  

3.2.2/ session driver interface

  maildriver is the driver structure for mail sessions

  - name is the name of the driver
  
  - initialize() is the function that will initializes a data structure
      specific to the driver, it returns a value that will be stored
      in the field data of the session.
      The field data of the session is the state of the session,
      the internal data structure used by the driver.
      It is called when creating the mailsession structure with
      mailsession_new().
  
  - uninitialize() frees the structure created with initialize()

  - parameters() implements functions specific to the given mail access
  
  - connect_stream() connects a stream to the session

  - connect_path() notify a main path to the session

  - starttls() changes the current stream to a TLS stream
  
  - login() notifies the user and the password to authenticate to the
      session

  - logout() exits the session and closes the stream

  - noop() does no operation on the session, but it can be
      used to poll for the status of the connection.

  - check_folder() makes a checkpoint of the session

  - select_folder() selects a mailbox

  - expunge_folder() deletes all messages marked \Deleted

  - status_folder() queries the status of the folder
      (number of messages, number of recent messages, number of
      unseen messages)

  - append_message() adds a RFC 2822 message to the current
      given mailbox

  - get_messages_list() returns the list of message numbers
      of the current mailbox.

  - get_envelopes_list() fills the parsed fields in the
      mailmessage structures of the mail_envelopes_list.

  - remove_message() removes the given message from the mailbox.
      The message is permanently deleted.

  - get_message returns a mailmessage structure that corresponds
      to the given message number.


3.2.3/ message driver interface

  mailmessage_driver is the driver structure to get information from messages.
  
  - name is the name of the driver

  - initialize() is the function that will initializes a data structure
      specific to the driver, it returns a value that will be stored
      in the field data of the mailsession.
      The field data of the session is the state of the session,
      the internal data structure used by the driver.
      It is called when initializing the mailmessage structure with
      mailmessage_init().
  
  - uninitialize() frees the structure created with initialize().
      It will be called by mailmessage_free().

  - flush() will free from memory all temporary structures of the message
      (for example, the MIME structure of the message).

  - fetch_result_free() will free all strings resulted by fetch() or
      any fetch_xxx() functions that returns a string.

  - fetch() returns the content of the message (headers and text).

  - fetch_header() returns the content of the headers.

  - fetch_body() returns the message text (message content without headers)

  - fetch_size() returns the size of the message content.

  - get_bodystructure() returns the MIME structure of the message.

  - fetch_section() returns the content of a given MIME part

  - fetch_section_header() returns the header of the message
      contained by the given MIME part.

  - fetch_section_mime() returns the MIME headers of the
      given MIME part.

  - fetch_section_body() returns the text (if this is a message, this is the
      message content without headers) of the given MIME part.

  - fetch_envelope() returns a mailimf_fields structure, with a list of
      fields chosen by the driver.

  - get_flags() returns a the flags related to the message.
      When you want to get flags of a message, you have to make sure to
      call get_flags() at least once before using directly message->flags.


3.3/ Higher level interface
---------------------------

3.3.1/ Files descriptions

generic_cache.[ch]              -- functions that implements cache and
                                     flags storing mechanism
imapdriver.[ch]                 -- IMAP driver for session
imapdriver_cached.[ch]          -- IMAP driver for session, using cache,
                                     IMAP already has flags.
imapdriver_cached_message.[ch]  -- IMAP driver for message, using cache
                                     IMAP already has flags.
imapdriver_message.[ch]         -- IMAP driver for message
imapdriver_types.[ch]           -- tools function for IMAP driver (types
                                     conversion from IMAP module).
imapstorage.[ch]                -- IMAP driver for storage
imfcache.[ch]                   -- implements cache for parsed fields
libetpan.h                      -- includes all necessary header files to
                                     use libEtPan!
maildriver.[ch]                 -- wrappers to calls to the session driver
maildriver_tools.[ch]           -- default implementation for drivers,
                                     when the driver does not parse the
                                     messages.
maildriver_types.[ch]           -- data types declaration and constructors
maildriver_types_helper.[ch]    -- easy data creation
mailmessage.[ch]                -- wrappers to calls to the message driver
mailstorage.[ch]                -- storage creation, calls to the storage
                                     driver and implementation of folders.
mailstorage_tools.[ch]          -- tools for storage (connection)
mailthread.[ch]                 -- threading: collection of the mails
                                     into a treee
mboxdriver.[ch]                 -- mbox driver for session
mboxdriver_cached.[ch]          -- mbox driver for session, using flags
                                     and cache
mboxdriver_cached_message.[ch]  -- mbox driver for message, using flags
                                     and cache
mboxdriver_message.[ch]         -- mbox driver for message
mboxdriver_tools.[ch]           -- mbox driver common functions
mboxstorage.[ch]                -- mbox driver for storage
mhdriver.[ch]                   -- MH driver for session
mhdriver_cached.[ch]            -- MH driver for session, using flags
                                     and cache
mhdriver_cached_message.[ch]    -- MH driver for message, using flags
                                     and cache.
mhdriver_message.[ch]           -- MH driver for message
mhdriver_tools.[ch]             -- MH driver common functions
mhstorage.[ch]                  -- MH driver for storage
nntpdriver.[ch]                 -- NNTP driver for session
nntpdriver_cached.[ch]          -- NNTP driver for session, using flags
                                     and cache
nntpdriver_cached_message.[ch]  -- NNTP driver for message, using flags
                                     and cache
nntpdriver_message.[ch]         -- NNTP driver for message
nntpdriver_tools.[ch]           -- NNTP driver common functions
nntpstorage.[ch]                -- NNTP driver for storage
pop3driver.[ch]                 -- POP3 driver for session
pop3driver_cached.[ch]          -- POP3 driver for session, using flags
                                     and cache
pop3driver_cached_message.[ch]  -- POP3 driver for message, using flags
                                     and cache
pop3driver_message.[ch]         -- POP3 driver for message
pop3driver_tools.[ch]           -- POP3 driver common functions
pop3storage.[ch]                -- POP3 driver for storage


3.3.2/ Interfaces

Include for this module is libetpan.h and includes all other headers.


The interface of higher layer is documented in the following files :

maildriver.h
maildriver_types.h
maildriver_types_helper.h
mailmessage.h
mailstorage.h
mailstorage_types.[h]
mailthread.h


4/ Architecture
---------------

(see layer.fig)


5/ Example of use
-----------------

You can find some example in tests/


6/ Constraints
--------------

- libEtPan! must run on a system where mmap() is available.

- for mbox particularly, libEtPan! make assumption on the fact that a
    file can be entirely mapped into memory. But if you don't read
	mailboxes of 1 Go, it should be fine.