Thursday, October 22

Show the complete apache config file

In the Apache config file, you can use "Include" or "IncludeOptional" to include other config files. A lot of the Linux variants take advantage of that to organize the config files. For example, the default congif file of Ubuntu is in /etc/apache2/apache.conf, and it includes enabled modules, enabled sites, and configuration files this way:

IncludeOptional mods-enabled/*.conf
IncludeOptional conf-enabled/*.conf
IncludeOptional sites-enabled/*.conf

If you really want to see all the complete config settings, there is no existing tool for that. This Stack Overflow page  answered this question pretty well: You can use apachectl -S to see the settings of Virtual Host, or apachectl -M to see the loaded modules, but to see all settings, there is no such tool, you will have to go through all the files , starting from familiar yourself with the  general structure of the httpd config files. 

So I created this python program to generate the complete apache config file:

__author__ = 'ben'
import sys, os, os.path, logging, fnmatch, re

def Help():
    print("Usage: python inputfile[default:/etc/apache2/apache2.conf] outputfile[default:/tmp/apache2.combined.conf")

def InputParameter():
    if len(sys.argv) <> 3:
        return "/etc/apache2/apache2.conf", "/tmp/apache2.combined.conf"
    return sys.argv[1], sys.argv[2]

def ProcessMultipleFiles(InputFiles):
    if InputFiles.endswith('/'):              #Updated as Pierrick's comment
        InputFiles = InputFiles + "*"
    Content = ''
    LocalFolder = os.path.dirname(InputFiles)
    basenamePattern = os.path.basename(InputFiles)
    for root, dirs, files in os.walk(LocalFolder):
        for filename in fnmatch.filter(files, basenamePattern):
            Content += ProcessInput(os.path.join(root, filename))
    return Content

def RemoveExcessiveLinebreak(s):
    Length = len(s)
    s = s.replace(os.linesep + os.linesep + os.linesep, os.linesep + os.linesep)
    NewLength = len(s)
    if NewLength < Length:
        s = RemoveExcessiveLinebreak(s)
    return s

def ProcessInput(InputFile):
    global ServerRoot

    Content = ''
    if logging.root.isEnabledFor(logging.DEBUG):
        Content = '# Start of ' + InputFile + os.linesep
    with open(InputFile, 'r') as infile:
        for line in infile:
            stripline = line.strip(' \t')
            if stripline.startswith('#'):
            searchroot ='ServerRoot\s+(\S+)', stripline, re.I)      #search for ServerRoot
            if searchroot:
                ServerRoot ='"')
      "ServerRoot: " + ServerRoot)
            if stripline.lower().startswith('include'):
                match = stripline.split()
                if len(match) == 2:
                    IncludeFiles = match[1]
                    IncludeFiles = IncludeFiles.strip('"') #Inserted according to V's comment.
                    if not IncludeFiles.startswith('/'):
                        IncludeFiles = os.path.join(ServerRoot, IncludeFiles)

                    Content += ProcessMultipleFiles(IncludeFiles) + os.linesep
                    Content += line     # if it is not pattern of 'include(optional) path', then continue.
                Content += line
    Content = RemoveExcessiveLinebreak(Content)
    if logging.root.isEnabledFor(logging.DEBUG):
        Content += '# End of ' + InputFile + os.linesep + os.linesep
    return Content

if __name__ == "__main__":
    logging.basicConfig(level=logging.DEBUG, format='[%(asctime)s][%(levelname)s]:%(message)s')
    InputFile, OutputFile = InputParameter()
        ServerRoot = os.path.dirname(InputFile)
        Content = ProcessInput(InputFile)
    except Exception as e:
        logging.error("Failed to process " + InputFile,  exc_info=True)

        with open(OutputFile, 'w') as outfile:
    except Exception as e:
        logging.error("Failed to write to " + outfile,  exc_info=True)
        exit(1)"Done writing " + OutputFile)

The usage is simple: Run it as "python ". Since there is no additional parameters given, it will retrieve the default Ubuntu apache config file from  /etc/apache2/apache2.conf and generate the result complete config file in /tmp/apache2.combined.conf. If your config file is in different location, then give the input file and output file location. For example, RHEL has the config file in /etc/httpd/conf/httpd.conf, then you can run "python /etc/httpd/conf/httpd.conf /tmp/apache2.combined.conf ".

Note: Apache server-info page also provide similar information, but not in the config file format. It is in human readable format. The page works only when it is open from the same computer.


Nice, but is there a way to dump the configs if macro_module is being used.

It's a nice looking idea, but doesn't seem to be including any included files... is this working for other people?
Let me correct/clarify that a bit... It seems to work if the include is a full path and not a relative path to the httpd config root (which works for apache and is allowed as far as I know).

This works
Include /etc/httpd/conf/vhosts.d/*.conf

This does not work (although apache is OK with it)
Include vhosts.d/*.conf

For some reason, this does not work (from a whm/cpanel httpd.conf)
Include "/etc/apache2/conf.d/userdata/std/2_4/username/*.conf"

I'm guessing the issue in the cpanel example might be the quotations around the string

Hi V,

Confirmed that path with quotations will not work. But the relative path definitely works well for me.

I will update the program to deal with quotations later. With source code, you should be able to do that as well :)

Inserted one line "IncludeFiles = IncludeFiles.strip('"')" to deal with the config file with quotation marks according to V's comment.

Thanks, V.
It does not work with folder includes. Like "Include conf.d/"
As workaround, I added the next lines in ProcessMultipleFiles:

if InputFiles.endswith('/'):
InputFiles = InputFiles + "*"

Also, the script may need a supplementary optional argument to specify relative path root. The script does not work on RHEL based systems. There, the base config path is /etc/httpd/conf/httpd.conf and root is /etc/httpd/.

Thank you, Pierrick. Confirmed that my version did not work with "Include conf.d/" scenario and your update was good, so I updated the post accordingly.

About the RHEL and optional relative path root, I will need that environment to try it out. Thanks.
Pierrick, after looking into an RHEL server, I found the reason was that my code ignored "ServerRoot" that Apache used to identify the root of config file ("relative path root" of your comment).

By adding the logic of ServerRoot (making a global variable, make it default as the folder of httpd.conf but available to be modified as reading the config file), the code is working well in both RHEL environment which has ServerRoot specified, and Debian environment.

Thanks for pointing out the issue!

After revisiting the previous comment, I think maybe Verdon's frustration come from the same source: the ServerRoot in RHEL environment was not handled properly so the relative path was not correctly processed.

Please use the new version!

Thursday, October 15

Python logging OnMarkRotatingFileHandler

When there is a need, there is a solution.

As explained in the previous post "The TimedRotatingFileHandler of python logging system", the handler is not doing what I am thinking to do. So I made this new handler OnMarkRotatingFileHandler to fulfill my need. For example, assuming the Interval setting is "Hour":

1, If the program starts at 8:20AM, the TimedRotatingFileHandler will restart a new log file at 9:20AM, but my OnMarkRotatingFileHandler will restart a new log file at 9:00AM.

2, If the previous log file was last modified at 8:58 AM, if you start the program with old handler after 9:58, it will rotate the log; if you start the program before 9:58, say 9:55 the program will just append the log entries into the existing log until 10:55, if your program runs for that long.
With my new handler, now at 9:00AM, when the program starts, the new handler will rotate the log file to generate new log file.

Source code:
__author__ = 'ben'

import logging.handlers

class OnMarkRotatingFileHandler(logging.handlers.TimedRotatingFileHandler):

        def __init__(self, filename, when='h', interval=1, backupCount=0, encoding=None, delay=False, utc=False):
            #super().__init__() # in Python 2 use super(D, self).__init__()
            super(OnMarkRotatingFileHandler, self).__init__(filename, when, interval, backupCount, encoding, delay, utc)

        def floor_to(self, num, scale):
            return int(num/scale) * scale

        def computeRollover(self, currentTime):
            temp_result = super(OnMarkRotatingFileHandler, self).computeRollover(currentTime)
            if not self.when.startswith('W'):
                result = self.floor_to(temp_result, self.interval)
                result = temp_result    # need to find out the first date of time (is it 1970/1/1?), what weekday that is.

            return result

Most methods are inherited from the TimedRotatingFileHandler. The W0/W1.../W6  options are not implemented yet. But you get the ideal.

To use it, place the with your code, then you can either import it and load it as

    import logHandler
    h = logHandler.OnMarkRotatingFileHandler ("filename")

or you can use it in the logging.ini:

then load it in logging.config like every other handler does:

Have fun hacking!


The TimedRotatingFileHandler of python logging system

The name of TimedRotatingFileHandler is like the best for my need: rotate every day, keep 30 days of copy. I know the logrotate of Linux system provides the same service, but I prefer to make the program easy to install and run without System Admin (root) involving in the process. Another reason I am not using logrotate is that when my program is writing into log at midnight, logrotate would perform "mv myprogram.log myprogram.log.20151015", and my program is still writing into that file which has a new name now: myprogram.log.20151015, while I still have another program trying to access myprogram.log for display purpose, then that program will fail.

So I want to have a native logging handler to do log rotating, and change to a new file of "myprogram.log" at midnight. The TimedRotatingFileHandler looks perfect to me.

But it is not.

The TimedRotatingFileHandler only rotate the log when the program has been running for more than 24 hours (Assuming the interval setting is 1 Day ), or if the log file has not been modified (before loading) for more than 24 hours. You can check the code, but that is what it is. If your program runs every day from 1AM to 11PM, then the log is never rotate, it keep appending forever. It is using the last modified time of the log file , or current time when the file doesn't exist, to decide when to rotate.

One limitation is that in Linux there is not "file creation time" or "birth time". Most file system don't support this file attribute. I guess ext4 does, under some configuration, but we can't rely on it.