Fork me on GitHub
GPyPi2 — Google Summer of Code week #7 and #8
20. Jul. 2010 at 14:49 written under portage, gentoo, python, gpypi2, gsoc, ebuild, english

Greetings, following is the weekly update for gpypi2 project. It's main purpose is to generate ebuilds from Python Package Index. Quick links to project info:

Very successful 2 weeks. Two days of vacations and nice shiney configuration utility together with questionnaire.

Previous 2 weeks (5th - 19th July)

Task: Write configuration skeleton

Currently, gpypi2 code only takes configuration from command line.
I'd like to extend that to config file (.ini), get data directly from setup.py
and interactive questionaire. In order for that to function properly,
configuration needs to be done in right sequence.

Following week I will write down the design and implement initial working prototype.

Configuration dispatcher fully implemented! Also we have already working questionnaire functionality. Details can be found at http://docs.fubar.si/gpypi2/userguide.html#configuration

Task: sync command - populate overlay with successful ebuilds from pypi

When I will get bored or stuck with task #1, I'll start working on command that will
generate ebuilds from all packages in pypi to an overlay.

Initial "sync" command is implemented. Still lacking proper error handling, metadata generation and manifest update support. Will be a lot easier to implement with current configuration management.

Additional, following mini-tasks were performed:

  • fixed echo command to work properly
  • correct handling of whitespace in rendered ebuilds
  • replace 4 spaces with a tab in rendered ebuilds

Upcoming week (20th - 26th July)

Task: fix TODOs in the code

Code currently has about `60 TODOs <http://hudson.fubar.si/job/g-pypi2/portage=portage2.1,python=2.6/58/tasksResult/?>`_ that need to be address for a better codebase.

Task: Implement "python setup.py bdist_ebuild" command

I'll peek into distutils heart and implement most basic bdist command to output an ebuild.
I will basically the same stuff we currently do,
except it will parse data from setup.py itself and not through pypi.
Read and Post Comments
Your personal, free webpage monitoring (powered by google app engine)
19. Jul. 2010 at 18:11 written under python, gae, monitoring, english

Thanks to http://xercestech.com/app-engine-server-monitor.geek

Go to https://appengine.google.com/ and register application with {app-name}

wget http://googleappengine.googlecode.com/files/google_appengine_1.3.5.zip
unzip google_appengine_1.3.5.zip
cd google_appengine
git clone git://github.com/mrsteveman1/aeservmon.git {app-name}
vim {app-name}/app.yaml # set application
python appcfg.py update {app-name}

Result: http://web-monitoring.appspot.com/

Read and Post Comments
Hidden gem in Python packaging world
07. Jul. 2010 at 17:04 written under python packaging setuptools distutils, english

Packaging love-hate world in Python is with every Python developer. Each time you need something from setuptools/distribute/distutils you actually find something new, useful. Can not deny that. There are quite some people working hard on distutils2 to remove the "miserable" word from packaging. I'm really hoping that lack of communication with Python developers and Linux distribution will be fixed when people will not only care, but also give constructive thoughts how distutils could be improved (and thus make life easier even to Linux distributions).

Enough of non-sense philosophy. Recently I found about "Feature" class in setuptools/distribute. Let's see what setuptools has to say about "features" keyword passed to the famous setup function:

'features' -- a dictionary mapping option names to 'setuptools.Feature'
        objects.  Features are a portion of the distribution that can be
        included or excluded based on user options, inter-feature dependencies,
        and availability on the current system.  Excluded features are omitted
        from all setup commands, including source and binary distributions, so
        you can create multiple distributions from the same source tree.
        Feature names should be valid Python identifiers, except that they may
        contain the '-' (minus) sign.  Features can be included or excluded
        via the command line options '--with-X' and '--without-X', where 'X' is
        the name of the feature.  Whether a feature is included by default, and
        whether you are allowed to control this from the command line, is
        determined by the Feature object.  See the 'Feature' class for more
        information.

Basically, you can do optional handling what should be installed when user executes "python setup.py --with-something install"

According to nullege, documentation for Feature class is the following:

A subset of the distribution that can be excluded if unneeded/wanted

Features are created using these keyword arguments:

  'description' -- a short, human readable description of the feature, to
     be used in error messages, and option help messages.

  'standard' -- if true, the feature is included by default if it is
     available on the current system.  Otherwise, the feature is only
     included if requested via a command line '--with-X' option, or if
     another included feature requires it.  The default setting is 'False'.

  'available' -- if true, the feature is available for installation on the
     current system.  The default setting is 'True'.

  'optional' -- if true, the feature's inclusion can be controlled from the
     command line, using the '--with-X' or '--without-X' options.  If
     false, the feature's inclusion status is determined automatically,
     based on 'availabile', 'standard', and whether any other feature
     requires it.  The default setting is 'True'.

  'require_features' -- a string or sequence of strings naming features
     that should also be included if this feature is included.  Defaults to
     empty list.  May also contain 'Require' objects that should be
     added/removed from the distribution.

  'remove' -- a string or list of strings naming packages to be removed
     from the distribution if this feature is *not* included.  If the
     feature *is* included, this argument is ignored.  This argument exists
     to support removing features that "crosscut" a distribution, such as
     defining a 'tests' feature that removes all the 'tests' subpackages
     provided by other features.  The default for this argument is an empty
     list.  (Note: the named package(s) or modules must exist in the base
     distribution when the 'setup()' function is initially called.)

  other keywords -- any other keyword arguments are saved, and passed to
     the distribution's 'include()' and 'exclude()' methods when the
     feature is included or excluded, respectively.  So, for example, you
     could pass 'packages=["a","b"]' to cause packages 'a' and 'b' to be
     added or removed from the distribution as appropriate.

A feature must include at least one 'requires', 'remove', or other
keyword argument.  Otherwise, it can't affect the distribution in any way.
Note also that you can subclass 'Feature' to create your own specialized
feature types that modify the distribution in other ways when included or
excluded.  See the docstrings for the various methods here for more detail.
Aside from the methods, the only feature attributes that distributions look
at are 'description' and 'optional'.

A very good example that abuses "features" is jinja2's setup.py.

Read and Post Comments
GPyPi2 — Google Summer of Code week #6
07. Jul. 2010 at 01:20 written under gentoo, python, gpypi2, english, gsoc, portage

Greetings, following is the weekly update for gpypi2 project. It's main purpose is to generate ebuilds from Python Package Index. Quick links to project info:

Last two weeks were crazy at university. Now I shall have time to work on gpypi2 full-time until the end of program.

Install gpypi2

$ sudo pip install http://bitbucket.org/iElectric/g-pypi2/get/tip.zip

or

$ sudo easy_install http://bitbucket.org/iElectric/g-pypi2/get/tip.zip

Create an ebuild

$ sudo gpypi2 create flask

 * Generating ebuild: Flask 0.4
 * Your ebuild is here: /usr/local/portage/dev-python/flask/flask-0.4.ebuild
 * Dependency needed: Werkzeug
 * Dependency needed: Jinja2
 * Generating ebuild: Werkzeug 0.6.2
 * Your ebuild is here: /usr/local/portage/dev-python/werkzeug/werkzeug-0.6.2.ebuild
 * Generating ebuild: Jinja2 2.5
 * Your ebuild is here: /usr/local/portage/dev-python/jinja2/jinja2-2.5.ebuild
 * Dependency needed: Babel
 * Generating ebuild: Babel 0.9.5
 * Your ebuild is here: /usr/local/portage/dev-python/babel/babel-0.9.5.ebuild

NOTE: ebuilds will probably not function yet, features are yet to be completed/added. Report bugs to `issue tracker <http://bitbucket.org/iElectric/g-pypi2/issues?status=new&status=open>`_.

Previous week (21st June - 5th July)

Task: Use SrcUriNamer

gpypi2.enamer.SrcUriNamer is awaiting for quite 2 weeks now
to be finally used.

It's basically factory that generates SRC_URI and HOMEPAGE
according to mirror provider.
A lot of additional refactoring will be needed to Enamer to
use this utility instead of current guessing/parsing.

Not done. With Jesus we decided to postpone this for one of the last tasks, since it's not that big benefit to gpypi2 and there are more important details to be implemented. It's also a big blocker for good unittest coverage, but that shall be sorted out.

Task: Fix bugs reported by users

Lot of bugs are going to pop up; fix bugs, write tests and
add documentation.

None reported :(

Task: Increase test coverage

I'm still not satisfied with test coverage.
I'll try to increase it a bit more.
It's 10 days week of exams on my university, soon I will be able to put
in more effort in implementing features.

Unittest coverage is about 75%. Won't get further than that until SrcUriNamer is properly implemented and tested. Few additional tests can be made at PortageUtils, but that's a minor issue. Here is a rundown of additional changes to code:

  • slight improvements to documentation
  • option to choose PyPi base URL
  • changed behaviour to use XML-RPC and SETUPTOOLS/SIMPLEPAGE method for retrieving SRC_URI
  • updated licences dictionary mapping
  • fixed echo command to work properly (without cleanup atm)

Upcoming week (22nd June - 28th June)

Task: Write configuration skeleton

Currently, gpypi2 code only takes configuration from command line.
I'd like to extend that to config file (.ini), get data directly from setup.py
and interactive questionaire. In order for that to function properly,
configuration needs to be done in right sequence.

Following week I will write down the design and implement initial working prototype.

Task: sync command - populate overlay with successful ebuilds from pypi

When I will get bored or stuck with task #1, I'll start working on command that will
generate ebuilds from all packages in pypi to an overlay.

Cheers, Domen

Read and Post Comments
GPyPi2 — Google Summer of Code week #4
21. Jun. 2010 at 17:55 written under gentoo, python, portage, english, gsoc, gpypi2

Greetings, following is the weekly update for gpypi2 project. It's main purpose is to generate ebuilds from Python Package Index. Quick links to project info:

I'm more than happy to announce first pre-alpha prototype. Intended goal is to make people start using the tool and reporting back their critics.

Install gpypi2

$ sudo pip install http://bitbucket.org/iElectric/g-pypi2/get/tip.zip

or

$ sudo easy_install http://bitbucket.org/iElectric/g-pypi2/get/tip.zip

Create an ebuild

$ sudo gpypi2 create flask

 * Generating ebuild: Flask 0.4
 * Your ebuild is here: /usr/local/portage/dev-python/flask/flask-0.4.ebuild
 * Dependency needed: Werkzeug
 * Dependency needed: Jinja2
 * Generating ebuild: Werkzeug 0.6.2
 * Your ebuild is here: /usr/local/portage/dev-python/werkzeug/werkzeug-0.6.2.ebuild
 * Generating ebuild: Jinja2 2.5
 * Your ebuild is here: /usr/local/portage/dev-python/jinja2/jinja2-2.5.ebuild
 * Dependency needed: Babel
 * Generating ebuild: Babel 0.9.5
 * Your ebuild is here: /usr/local/portage/dev-python/babel/babel-0.9.5.ebuild

NOTE: ebuilds will probably not function yet, features are yet to be completed/added. Report bugs to `issue tracker <http://bitbucket.org/iElectric/g-pypi2/issues?status=new&status=open>`_.

Previous week (15-21 June)

Main goal of previous week was to get closer to first prototype. Done.

Task: First snippet of command line interface

Put enough together to assemble ebuilds from command line. I think
many issues may pop up, and lots of bugs shall be fixed. I'm currently
mostly doing stuff from my head, with not much practise whatsoever.

Mostly, we should get a result of better test coverage (currrently about
50%).

The only component left untouched is the configuration. I would like to
touch this topic after all my exams are over, leaving me time and
productivity.

Command line interface was added, test coverage is up to 60%. Lots of small fixes were done. As a bonus, I added portage output syntax colors.

Task: get_dependencies - another refactor to enamer

Title says it all. Also handling of multiple version operators should be
added.

Example: foobar>=0.5&& foobar<0.6

should resolve in <0.6 blocker and >=0.5 requirement.

Multiple operators on version dependencies are handled as blockers, implemented in tip. However get_dependencies was not refactored, since I still might be adding functionality. Not really necessary for now.

Upcoming week (22-28 June)

Task: Use SrcUriNamer

gpypi2.enamer.SrcUriNamer is awaiting for quite 2 weeks now
to be finally used.

It's basically factory that generates SRC_URI and HOMEPAGE
according to mirror provider.
A lot of additional refactoring will be needed to Enamer to
use this utility instead of current guessing/parsing.

Task: Fix bugs reported by users

Lot of bugs are going to pop up; fix bugs, write tests and
add documentation.

Task: Increase test coverage

I'm still not satisfied with test coverage.
I'll try to increase it a bit more.
It's 10 days week of exams on my university, soon I will be able to put
in more effort in implementing features.

Cheers, Domen

Read and Post Comments