Show the complete apache config file

In the Apache config file, you can use "Include" or "IncludeOptional" to include other config files. A lot of the Linux variants take advantage of that to organize the config files. For example, the default congif file of Ubuntu is in /etc/apache2/apache.conf, and it includes enabled modules, enabled sites, and configuration files this way:

IncludeOptional mods-enabled/*.conf
IncludeOptional conf-enabled/*.conf
IncludeOptional sites-enabled/*.conf

If you really want to see all the complete config settings, there is no existing tool for that. This Stack Overflow page  answered this question pretty well: You can use apachectl -S to see the settings of Virtual Host, or apachectl -M to see the loaded modules, but to see all settings, there is no such tool, you will have to go through all the files , starting from familiar yourself with the  general structure of the httpd config files. 

So I created this python program to generate the complete apache config file:

#!/usr/bin/python2.7
# CombineApacheConfig.py 

#!/usr/bin/python2.7
# CombineApacheConfig.py 
__author__ = 'ben'
import sys, os, os.path, logging, fnmatch, re


def Help():
    print("Usage: python CombineApacheConfig.py inputfile[default:/etc/apache2/apache2.conf] outputfile[default:/tmp/apache2.combined.conf")


def InputParameter():
    if len(sys.argv) <> 3:
        Help()
        return "/etc/apache2/apache2.conf", "/tmp/apache2.combined.conf"
    return sys.argv[1], sys.argv[2]


def ProcessMultipleFiles(InputFiles):
    if InputFiles.endswith('/'):              #Updated as Pierrick's comment
        InputFiles = InputFiles + "*"
    Content = ''
    LocalFolder = os.path.dirname(InputFiles)
    basenamePattern = os.path.basename(InputFiles)
    for root, dirs, files in os.walk(LocalFolder):
        for filename in fnmatch.filter(files, basenamePattern):
            Content += ProcessInput(os.path.join(root, filename))
    return Content


def RemoveExcessiveLinebreak(s):
    Length = len(s)
    s = s.replace(os.linesep + os.linesep + os.linesep, os.linesep + os.linesep)
    NewLength = len(s)
    if NewLength < Length:
        s = RemoveExcessiveLinebreak(s)
    return s


def ProcessInput(InputFile):
    global ServerRoot

    Content = ''
    if logging.root.isEnabledFor(logging.DEBUG):
        Content = '# Start of ' + InputFile + os.linesep
    with open(InputFile, 'r') as infile:
        for line in infile:
            stripline = line.strip(' \t')
            if stripline.startswith('#'):
                continue
            searchroot = re.search(r'ServerRoot\s+(\S+)', stripline, re.I)      #search for ServerRoot
            if searchroot:
                ServerRoot = searchroot.group(1).strip('"')
                logging.info("ServerRoot: " + ServerRoot)
            if stripline.lower().startswith('include'):
                match = stripline.split()
                if len(match) == 2:
                    IncludeFiles = match[1]
                    IncludeFiles = IncludeFiles.strip('"') #Inserted according to V's comment.
                    if not IncludeFiles.startswith('/'):
                        IncludeFiles = os.path.join(ServerRoot, IncludeFiles)

                    Content += ProcessMultipleFiles(IncludeFiles) + os.linesep
                else:
                    Content += line     # if it is not pattern of 'include(optional) path', then continue.
            else:
                Content += line
    Content = RemoveExcessiveLinebreak(Content)
    if logging.root.isEnabledFor(logging.DEBUG):
        Content += '# End of ' + InputFile + os.linesep + os.linesep
    return Content


if __name__ == "__main__":
    logging.basicConfig(level=logging.DEBUG, format='[%(asctime)s][%(levelname)s]:%(message)s')
    InputFile, OutputFile = InputParameter()
    try:
        ServerRoot = os.path.dirname(InputFile)
        Content = ProcessInput(InputFile)
    except Exception as e:
        logging.error("Failed to process " + InputFile,  exc_info=True)
        exit(1)

    try:
        with open(OutputFile, 'w') as outfile:
            outfile.write(Content)
    except Exception as e:
        logging.error("Failed to write to " + outfile,  exc_info=True)
        exit(1)

    logging.info("Done writing " + OutputFile)

The usage is simple: Run it as "python  CombineApacheConfig.py ". Since there is no additional parameters given, it will retrieve the default Ubuntu apache config file from  /etc/apache2/apache2.conf and generate the result complete config file in /tmp/apache2.combined.conf. If your config file is in different location, then give the input file and output file location. For example, RHEL has the config file in /etc/httpd/conf/httpd.conf, then you can run "python CombineApacheConfig.py /etc/httpd/conf/httpd.conf /tmp/apache2.combined.conf ".

Note: Apache server-info page http://127.0.0.1/server-info also provide similar information, but not in the config file format. It is in human readable format. The page works only when it is open from the same computer.

Python logging OnMarkRotatingFileHandler

When there is a need, there is a solution.

As explained in the previous post "The TimedRotatingFileHandler of python logging system", the handler is not doing what I am thinking to do. So I made this new handler OnMarkRotatingFileHandler to fulfill my need. For example, assuming the Interval setting is "Hour":

1, If the program starts at 8:20AM, the TimedRotatingFileHandler will restart a new log file at 9:20AM, but my OnMarkRotatingFileHandler will restart a new log file at 9:00AM.

2, If the previous log file was last modified at 8:58 AM, if you start the program with old handler after 9:58, it will rotate the log; if you start the program before 9:58, say 9:55 the program will just append the log entries into the existing log until 10:55, if your program runs for that long.
With my new handler, now at 9:00AM, when the program starts, the new handler will rotate the log file to generate new log file.


Source code:

#filename: logHandler.py
__author__ = 'ben'

import logging.handlers

class OnMarkRotatingFileHandler(logging.handlers.TimedRotatingFileHandler):

        def __init__(self, filename, when='h', interval=1, backupCount=0, encoding=None, delay=False, utc=False):
            #super().__init__() # in Python 2 use super(D, self).__init__()
            super(OnMarkRotatingFileHandler, self).__init__(filename, when, interval, backupCount, encoding, delay, utc)


        def floor_to(self, num, scale):
            return int(num/scale) * scale


        def computeRollover(self, currentTime):
            temp_result = super(OnMarkRotatingFileHandler, self).computeRollover(currentTime)
            if not self.when.startswith('W'):
                result = self.floor_to(temp_result, self.interval)
            else:
                result = temp_result    # need to find out the first date of time (is it 1970/1/1?), what weekday that is.

            return result

Most methods are inherited from the TimedRotatingFileHandler. The W0/W1.../W6  options are not implemented yet. But you get the ideal.

To use it, place the logHandler.py with your code, then you can either import it and load it as

    import logHandler
    h = logHandler.OnMarkRotatingFileHandler ("filename")
    logger=logging.getLogger('app')
    logger.addHandler(h)

or you can use it in the logging.ini:

    class=logHandler.OnMarkRotatingFileHandler

then load it in logging.config like every other handler does:

    logging.config.fileConfig('logging.ini')

Have fun hacking!

The TimedRotatingFileHandler of python logging system

The name of TimedRotatingFileHandler is like the best for my need: rotate every day, keep 30 days of copy. I know the logrotate of Linux system provides the same service, but I prefer to make the program easy to install and run without System Admin (root) involving in the process. Another reason I am not using logrotate is that when my program is writing into log at midnight, logrotate would perform "mv myprogram.log myprogram.log.20151015", and my program is still writing into that file which has a new name now: myprogram.log.20151015, while I still have another program trying to access myprogram.log for display purpose, then that program will fail.


So I want to have a native logging handler to do log rotating, and change to a new file of "myprogram.log" at midnight. The TimedRotatingFileHandler looks perfect to me.


But it is not.


The TimedRotatingFileHandler only rotate the log when the program has been running for more than 24 hours (Assuming the interval setting is 1 Day ), or if the log file has not been modified (before loading) for more than 24 hours. You can check the code, but that is what it is. If your program runs every day from 1AM to 11PM, then the log is never rotate, it keep appending forever. It is using the last modified time of the log file , or current time when the file doesn't exist, to decide when to rotate.


One limitation is that in Linux there is not "file creation time" or "birth time". Most file system don't support this file attribute. I guess ext4 does, under some configuration, but we can't rely on it.