Differences between revisions 1 and 21 (spanning 20 versions)
Revision 1 as of 2004-01-12 10:23:19
Size: 1593
Editor: cache1-1-ffm-vpn
Comment:
Revision 21 as of 2005-08-24 21:49:34
Size: 7028
Editor: 208
Comment: Added example py2exe subclass to support automatic UPXing of dll/pyd modules
Deletions are marked like this. Additions are marked like this.
Line 3: Line 3:
within samples/extending Thomas Heller gives a setup.py scipt which automagically calls [http://www.jrsoftware.org/isinfo.php InnoSetup] to create a great looking (and perfectly working) Windows-Installer. Yes, really, ONE setup.exe is created which contains ALL stuff to have a readily installable Software out of your product. Within samples/extending, Thomas Heller gives a Py2Exe setup.py script which automagically calls [http://www.jrsoftware.org/isinfo.php InnoSetup] to create a great looking (and perfectly working) Windows-Installer. Yes, really, ONE setup.exe is created which contains ALL stuff to have a readily installable Software out of your product.
Line 7: Line 7:
I programm using wxPython. A great library, a large library. I program using wxPython. A great library, a large library.
Line 9: Line 9:
   2.240.512 wxc.pyd
   3.416.064 wxmsw24uh.dll
{{{
2.240.512 wxc.pyd 
3.416.064 wxmsw24uh.dll
}}}
Line 12: Line 14:
To modules you cannnot skip. In addition there is as allways Two modules you cannnot skip. In addition there is as allways
   {{{
974.915 python23.dll
}}}
Line 14: Line 20:
    974.915 python23.dll

giving more than 6 Megabyte with no line of code written
giving more than 6 Megabytes with no line of code written
Line 20: Line 24:
    28 Datei(en) 9.549.525 Bytes (in lib) {{{
28 Datei(en) 9.549.525 Bytes (in lib)
}}}
Line 22: Line 28:
     2 Datei(en) 1.011.779 Bytes (in prog) {{{
2 Datei(en) 1.011.779 Bytes (in prog)
}}}
Line 33: Line 41:
        print >> ofi, r"[Setup]"
        print >> ofi, r"AppName=%s" % self.name
        print >> ofi, r"AppVerName=%s %s" % (self.name, self.version)
        print >> ofi, r"DefaultDirName=c:\ghum\%s" % self.name
        
print >> ofi, r"DefaultGroupName=%s" % self.name
        print >> ofi, r"Compression=bzip"
        print >> ofi
{{{
#!python
print >> ofi, r"[Setup]"
print >> ofi, r"AppName=%s" % self.name
print >> ofi, r"AppVerName=%s %s" % (self.name, self.version)
print >> ofi, r"DefaultDirName={pf}\%s" % self.name
print >> ofi, r"DefaultGroupName=%s" % self.name
print >> ofi, r"Compression=bzip"
print >> ofi
}}}
Line 44: Line 54:
   print >> ofi, r"Compression=bzip" {{{
#!python
print >> ofi, r"Compression=bzip"
}}}
Line 49: Line 62:
    4.335.283 setup.exe {{{
4.335.283 setup.exe
}}}
Line 54: Line 69:
= Remarks =
Line 55: Line 71:
 * I ([http://videocapture.sourceforge.net/ MGr]) get for my application (which uses wxWindows) even smaller files when using 7-zip instead of bzip. To use 7-zip one has to specify "Compression=none" in the generated installer script (*.iss) and then manually load the script into [http://www.istool.org/ ISTool], which is a front end to Inno Setup. Clicking at the 'Options' button and selecting '7-zip' in the compression combo box does the trick.{{{
7.208 kB uncompressed
2.886 kB zip
2.677 kB bzip
2.160 kB 7-zip
}}}The previous comparison shows, that 7-zip results by far in the smallest installer size. An additional benefit of using 7-zip is the fast installation. When using bzip, the files are decompressed in the last step of the installation process, right then when the files are written to the program folder. Since bzip is computationally quite intensive, the progress bar fills not extremely fast. When using 7-zip, there is a short decompression when starting the installer, but the installation process afterwards it fast as lightning, because of "Compression=none" :) .
Line 56: Line 78:
Addition:
Line 57: Line 80:
Innosetup 4.2 has 7z "LZMA" compression included. Just use
Line 58: Line 82:
{{{
#!python
print >> ofi, r"Compression=lzma/max"
}}}
Line 59: Line 87:
HAM2005-05-20
= 7ZIP and UPX =
Repacking the "library.zip" with [http://www.7-zip.org/ 7-ZIP] saving 357KB (577KB original to 220KB 7ZIP compressed).
Note: it requires "import zlib" in your source file to allow unzipping operations.

Using [http://upx.sourceforge.net/ UPX ] the executable packer to compress *.pyd, *.dll and *.exe Files:

Example: _sre.pyd, _winreg.pyd, python23.dll, unicodedata.pyd, w9xpopen.exe, zlib.pyd 1516,2KB original to 584KB compressed)

Everything together: Original Size: 2093,2 compressed: 804KB !!!

A short Windows Batch-File to automate (see WinBatch):
{{{
7z.exe -aoa x library.zip -olibrary\
del library.zip

cd library\
7z.exe a -tzip -mx9 ..\library.zip -r
cd..
rd library /s /q

upx.exe --best *.*
}}}

= Automatically UPXing extension modules/dlls =

While converting over from prior use of the Installer package, I missed its ability to automatically UPX binary modules as they were included in the distribution. So this example creates a custom command class based on py2exe that performs the UPX as part of the file copy process into the distribution directory. The one difference compared to Installer is that it doesn't currently keep a cache of UPXed files, although it could easily enough be extended to do that.

Note that as designed I believe it will UPX any dll/pyd file copied over, even if they are part of the data files for the application. But since anything packaged up with py2exe is going to be running independently and unlikely to need to share any DLLs with other applications, there's little harm in UPXing anything in sight.

The class below is a portion of a py2exe-subclass command class that I use that does the following, based on a new "upx" configuration value:

 * Compress (using upx) any dll or pyd file that is copied into the distribution directory. Since it does this at copy time, you only take the hit of upx whenever the file is updated (or you clean out the dist directory).
 * Deals with the fact that you have to postpone UPXing the main python##.dll file until py2exe has a chance to fix it's version identification, and that you shouldn't re-patch the file once it's been UPXed.

This command class was written with py2exe 0.5.4.

{{{
from distutils.core import setup
from py2exe.build_exe import py2exe

#--------------------------------------------------------------------------
#
# Define our own command class based on py2exe so we can perform some
# customizations, and in particular support UPXing the binary files.
#
#--------------------------------------------------------------------------

class Py2exe(py2exe):

    def initialize_options(self):
        # Add a new "upx" option for compression with upx
        py2exe.initialize_options(self)
        self.upx = 0

    def copy_file(self, *args, **kwargs):
        # Override to UPX copied binaries.
        (fname, copied) = result = py2exe.copy_file(self, *args, **kwargs)

        basename = os.path.basename(fname)
        if (copied and self.upx and
            (basename[:6]+basename[-4:]).lower() != 'python.dll' and
            fname[-4:].lower() in ('.pyd', '.dll')):
            os.system('upx --best "%s"' % os.path.normpath(fname))
        return result

    def patch_python_dll_winver(self, dll_name, new_winver=None):
        # Override this to first check if the file is upx'd and skip if so
        if not self.dry_run:
            if not os.system('upx -qt "%s" >nul' % dll_name):
                if self.verbose:
                    print "Skipping setting sys.winver for '%s' (UPX'd)" % \
                          dll_name
            else:
                py2exe.patch_python_dll_winver(self, dll_name, new_winver)
                # We UPX this one file here rather than in copy_file so
                # the version adjustment can be successful
                if self.upx:
                    os.system('upx --best "%s"' % os.path.normpath(dll_name))
}}}

This class would then be used in your main setup() call as:

{{{
setup(...
      cmdclass = {'py2exe': Py2exe}
      ...
      )
}}}

-- David Bolen

Something great

Within samples/extending, Thomas Heller gives a Py2Exe setup.py script which automagically calls [http://www.jrsoftware.org/isinfo.php InnoSetup] to create a great looking (and perfectly working) Windows-Installer. Yes, really, ONE setup.exe is created which contains ALL stuff to have a readily installable Software out of your product.

also large

I program using wxPython. A great library, a large library.

2.240.512 wxc.pyd 
3.416.064 wxmsw24uh.dll

Two modules you cannnot skip. In addition there is as allways

974.915 python23.dll

giving more than 6 Megabytes with no line of code written

In total:

28 Datei(en)      9.549.525 Bytes  (in lib)

and

2 Datei(en)      1.011.779 Bytes   (in prog)

summing up to 10.5 Megabyte ... not much with DSL, but... shocking, isn't it?

get it smaller

In the standard configuration [http://www.jrsoftware.org/isinfo.php InnoSetup] compressed it down to around 6.5 Megabyte.

I changed the setup.py to contain

   1 print >> ofi, r"[Setup]"
   2 print >> ofi, r"AppName=%s" % self.name
   3 print >> ofi, r"AppVerName=%s %s" % (self.name, self.version)
   4 print >> ofi, r"DefaultDirName={pf}\%s" % self.name
   5 print >> ofi, r"DefaultGroupName=%s" % self.name
   6 print >> ofi, r"Compression=bzip" 
   7 print >> ofi

... the special line is

   1 print >> ofi, r"Compression=bzip" 

the result

4.335.283 setup.exe 

Quite an OK download ... and I use ZODB and Elementtree and pythoncom and and and....

Remarks

  • I ([http://videocapture.sourceforge.net/ MGr]) get for my application (which uses wxWindows) even smaller files when using 7-zip instead of bzip. To use 7-zip one has to specify "Compression=none" in the generated installer script (*.iss) and then manually load the script into [http://www.istool.org/ ISTool], which is a front end to Inno Setup. Clicking at the 'Options' button and selecting '7-zip' in the compression combo box does the trick.

    7.208 kB  uncompressed
    2.886 kB  zip
    2.677 kB  bzip
    2.160 kB  7-zip

    The previous comparison shows, that 7-zip results by far in the smallest installer size. An additional benefit of using 7-zip is the fast installation. When using bzip, the files are decompressed in the last step of the installation process, right then when the files are written to the program folder. Since bzip is computationally quite intensive, the progress bar fills not extremely fast. When using 7-zip, there is a short decompression when starting the installer, but the installation process afterwards it fast as lightning, because of "Compression=none" :) .

Addition:

Innosetup 4.2 has 7z "LZMA" compression included. Just use

   1 print >> ofi, r"Compression=lzma/max" 

HAM2005-05-20

7ZIP and UPX

Repacking the "library.zip" with [http://www.7-zip.org/ 7-ZIP] saving 357KB (577KB original to 220KB 7ZIP compressed). Note: it requires "import zlib" in your source file to allow unzipping operations.

Using [http://upx.sourceforge.net/ UPX ] the executable packer to compress *.pyd, *.dll and *.exe Files:

Example: _sre.pyd, _winreg.pyd, python23.dll, unicodedata.pyd, w9xpopen.exe, zlib.pyd 1516,2KB original to 584KB compressed)

Everything together: Original Size: 2093,2 compressed: 804KB !!!

A short Windows Batch-File to automate (see WinBatch):

7z.exe -aoa x library.zip -olibrary\
del library.zip

cd library\
7z.exe a -tzip -mx9 ..\library.zip -r
cd..
rd library /s /q

upx.exe --best *.*

Automatically UPXing extension modules/dlls

While converting over from prior use of the Installer package, I missed its ability to automatically UPX binary modules as they were included in the distribution. So this example creates a custom command class based on py2exe that performs the UPX as part of the file copy process into the distribution directory. The one difference compared to Installer is that it doesn't currently keep a cache of UPXed files, although it could easily enough be extended to do that.

Note that as designed I believe it will UPX any dll/pyd file copied over, even if they are part of the data files for the application. But since anything packaged up with py2exe is going to be running independently and unlikely to need to share any DLLs with other applications, there's little harm in UPXing anything in sight.

The class below is a portion of a py2exe-subclass command class that I use that does the following, based on a new "upx" configuration value:

  • Compress (using upx) any dll or pyd file that is copied into the distribution directory. Since it does this at copy time, you only take the hit of upx whenever the file is updated (or you clean out the dist directory).
  • Deals with the fact that you have to postpone UPXing the main python##.dll file until py2exe has a chance to fix it's version identification, and that you shouldn't re-patch the file once it's been UPXed.

This command class was written with py2exe 0.5.4.

from distutils.core import setup
from py2exe.build_exe import py2exe

#--------------------------------------------------------------------------
#
# Define our own command class based on py2exe so we can perform some
# customizations, and in particular support UPXing the binary files.
#
#--------------------------------------------------------------------------

class Py2exe(py2exe):

    def initialize_options(self):
        # Add a new "upx" option for compression with upx
        py2exe.initialize_options(self)
        self.upx = 0

    def copy_file(self, *args, **kwargs):
        # Override to UPX copied binaries.
        (fname, copied) = result = py2exe.copy_file(self, *args, **kwargs)

        basename = os.path.basename(fname)
        if (copied and self.upx and
            (basename[:6]+basename[-4:]).lower() != 'python.dll' and
            fname[-4:].lower() in ('.pyd', '.dll')):
            os.system('upx --best "%s"' % os.path.normpath(fname))
        return result

    def patch_python_dll_winver(self, dll_name, new_winver=None):
        # Override this to first check if the file is upx'd and skip if so
        if not self.dry_run:
            if not os.system('upx -qt "%s" >nul' % dll_name):
                if self.verbose:
                    print "Skipping setting sys.winver for '%s' (UPX'd)" % \
                          dll_name
            else:
                py2exe.patch_python_dll_winver(self, dll_name, new_winver)
                # We UPX this one file here rather than in copy_file so
                # the version adjustment can be successful
                if self.upx:
                    os.system('upx --best "%s"' % os.path.normpath(dll_name))

This class would then be used in your main setup() call as:

setup(...
      cmdclass = {'py2exe': Py2exe}
      ...
      )

-- David Bolen

BetterCompression (last edited 2008-07-08 11:27:44 by localhost)