python technologies

[PyTI] PyPI Testing Infrastructure (summary)

2010-08-16T09:22:00.010Z

Hello All,

the GSoC finish today, so i just wanted to give you a update of the status of the project that i have been working on during this last 3 mount, but first of all i want to thanks all the people that help me to accomplish this work specially my mentor Jesse Noller for all his advices and remarks , and i want to thanks all the team of packagers that you can found on this link.

As usual the project can be found on the repository where you can find more information about the project , how to install the application.

An other thing the project is already deployed on : PyTI .

For the issues and bugs in the project check also : issues and bugs.

About all the weekly report that we did have to write in every week check : ML.

How it's work ?

you can follow the How to install to get the application ready.

now after you did run PyTI, go to your terminal and do:

cd pony-build/client
python build-pypi-package.py -s

This command call the build-pypi-package.py pony-build client that will build the package specify and send the result to the server to be displayed on the web interface.

For the build it's follow 3 steps:
- download the package from pypi
- run tox on the package : this will try to install the package using py26 and will run all package tests using py.tests
- finally it's call cheesecake to calculate the kwality factors of the package given .

here is some demo picture :

- after installing PyTI, i run the server : cd GUI/ ; python run.py --port 8800
- now i build a package i chose tox : python build-pypi-package.py -s http://127.0.0.1:8800 tox
- after the client finish building the package, i enter in my web browser http://127.0.0.1:8800
the first thing i see is a list of all build packages, i remark also that tox build failed
with the red image in the right.

- after clicking on the package you will see the detail of the build, first of all you will notice a time line of builds, the red circle is for failed build:

- you can choose a build event on the time line and click on it you will a pop up with some detail on it and a link to see full build report

- after clicking on the "see full report" link the report get displayed to you , with tree tabs, one for the download command , the other for tox and the last one for cheesecake , the ones in red mean that the command have failed in this case tox have failed

- now you can go to the tab tox to see why the command failed , in this case we will notice that 2 tests have failed from 74 tests run

- last you can go to the cheesecake tab to see the cheesecake report in this case tox have scored 59% :

All the work that have been done

this is a summarizing of all the work that have been done :

- add EC2Context to pony-build client that enable pony-build to run tests in an EC2 machine
- preparing an EC2 machine where package get build
- create a client that build package from pypi.
- create an adapter to get a django project work with pony-build
- create a django project with several application for displaying information on build packages
- one of the django application is used to get PuSH notifications from pypi to build the packages but unfortunately this has some issues on it .

Peace,
Mouad Benchchaoui

2010-07-13T12:55:00.002+01:00

Hello All,

while continuing working on the PyPI Testing Infrastructure (PyTI)[0], this week i will work on adding an EC2 interface to pony-build which will enable us to run package test in the EC2 Virtual Machine from the pony-build server.

The idea of using the EC2 machine was in the first place because we wanted to run test in a clean environment to do so Tarek Ziade has proposed to run test in an EC2 machine[1] after we reset the machine like that we have always a clean environment where we run tests .

About how to implement and after the suggestion of my mentor Jesse Noller to use server provisioning here is my ideas until now :

The problem :

- EC2 machines are in the first place down and i have to init them

- i have to follow the pony-build architecture which is client - server (so no master slave, so commanding client from server).

- the EC2 instance are fractured by the hour so when i finish testing i have to put it down again.

Idea:

The questions that give me the answer was :

is it an obligation that the test client (script) must be run in the test platform (EC2 VM) ?

The answer was : no, what should be run in the test platform is the test commands .

So the idea is to run the test client in a machine (in my case and to simplify it's will be the same where i run the server ) , it's will be like a Hub and from their i init the VM and run all commands and then i can put the VM down again.

To do so , i will create first a "Context" in pony-build like the "virtualenvContext" in which i will init and finish my EC2 VM and for running commands i will have to change the _run_commands function to enable it to run commands from a remote machine .

i think it's the easy way to do it the first time, if any one have another idea please let me know .

[0] http://bitbucket.org/mouad/pypi-testing-infrastructure-pyti/
[1] http://tarekziade.wordpress.com/2010/03/21/another-gsoc-idea-a-pypi-testing-infrastructure/

Creating a new web interface for pony-build

2010-07-01T18:53:00.002+01:00

Hello ,

In this tutorial i will try to explain the procedure that i follow to create a web interface for pony-build [0] which is a continuous integration system that i'm using in the GSoC project "pypi testing infrastructure"[1].

So first pony-build come with a default web interface created with Quixote and if you want to create a new web interface let say using Django in this tutorial i will show you how i interface the my web interface with pony-build .

1- First you have to run pony-build using the web interface in my project i created a script that i have called run.py where i have put this code:

from django.core.handlers.wsgi import WSGIHandler
from django.core.servers.basehttp import AdminMediaHandler

from adapter import CoordinatorAdapter
from pony_build import server, coordinator, dbsqlite

if __name__ == '__main__':
""" run pony-build server in local host

"""
_host = 'localhost'
_port = 8800

# create the sqlite database
_db_file = dbsqlite.open_shelf("pyti")
_db_file = coordinator.IntDictWrapper(_db_file)

# pass the database to the coordinator
my_coordinator = CoordinatorAdapter(db=_db_file)

# create the server using server module
pony_server = server.create(_host, _port, my_coordinator, AdminMediaHandler(WSGIHandler()))

print 'pony-build server started on host %s, port %d' % (_host, _port)

# run the server
pony_server.serve_forever()

2- Next i created a new implementation of the coordianator in my case i need it because i have two database the first one sqlite is used by the pony-build server to save packages build info and the other is used by Django , all this complexity is because i want to change the database layer to support not only sqlite and like that i will have only one database but not for now .

here my code :

import datetime
from django.contrib.auth.models import User

from pony_build.coordinator import PonyBuildCoordinator
from packages.models import Packages, History, Report

# inherit from the PonyBuildCoordinator
class CoordinatorAdapter(PonyBuildCoordinator):
""" Adapter design pattern

"""

# override add_result method so i can save the build info in the web interface DB
def add_results(self, client_ip, client_info, results):
""" copy build result to Django database.

"""

super(CoordinatorAdapter, self).add_results(client_ip, client_info, results)

#update existing package info or creating a new entry
new = Packages.objects.get_or_create(package_name=client_info['package'])[0]

new.package_description=client_info['package']
new.last_state = client_info['success']
new.last_build = datetime.datetime.now()
new.save()

package_history = History()

package_history.time = new.last_state
package_history.state = new.last_build
package_history.package_build = new

new_report = Report()
new_report.platform = client_info['arch']
new_report.report = 'TO ADD'
new_report.save()

package_history.report = new_report
package_history.save()

print "package saved"

if you want to see all the example go to [2]

[0] http://github.com/ctb/pony-build
[1] http://bitbucket.org/mouad/pypi-testing-infrastructure-pyti/
[2] http://bitbucket.org/mouad/pypi-testing-infrastructure-pyti/src/tip/GUI/

PYTI report #3

2010-06-24T18:49:00.002+01:00

Hello All,

The other week i didn't have time to write in the blog it's my last year in my school and it's the period of last exams so i was very busy preparing, i still have to do some stuff for my school until this Saturday, but here what i was doing for PYTI when i did have time for it.

Now i'm working on changing the database layer because when i first ran the pony-build test i did have some problem with one of them and because i'm using postgresql to save packages i have to change the database layer of pony that support only sqlite and i think that it's the problem for not having test succed, so
i want to make sure that all tests passed before adding something to pony-buid.

So first thing when i will finish the database layer and if all test pass i will start with the ec2 interface, i will make first a prototype because in the first i wanted to change the pony-build structure to make it more modular so that the pony-build interface can be just a module that you can enable but now i think that i'm late from my initial schedule and i didn't start yet the virtual machine part and i already discuss with Titus about my ideas for the pony-build and like he said maybe it's better to make it work first with necessary and minimal change then trying to change or add other functionalities.

http://bitbucket.org/mouad/pypi-testing-infrastructure-pyti

GSOC second meeting: PYPI Testing Infrastructure

2010-06-10T18:24:00.009+01:00

After finishing the first steps of the web interface, now i have to integrate the web interface with the pony-build server.

For that matter pony-build have a wsgi interface that i'm trying to use but with some difficulties, hopefully i will overpass.

Also i'm thinking about a way to use the coordinator.py (API for pony-build) using an adapter pattern to interface the django application with the pony server , i'm thinking about changing the database layer because dealing with two databases will be useless (pony build work only with sqlite).

Finally i'm thinking also about making pony-build configurable by adding a configuration file where you can define the database, choose the web interface , choose the notification way ....

this week was my first real contact with pony-build and i think that it's a very good project, with a lot of documentation, and testing (so that i can play with it safely)

To close this post i will display the first screen shot of the web interface that i spend the last weeks doing, it's still need some work and every comment is welcome.

- login page support "openid" from different account like shown in the picture, i think the openid must be optional that user can disable (i'm talking about running the web interface locally)
- After the authentication the user are redirect to a page where he can see his package with some information like the last build date, the status of the package (the status of the last build) , and he can choose from the tool bar above the action that he want to do like configure the server, see package detail , build package

You can follow as on this link for more information :
http://bitbucket.org/mouad/pypi-testing-infrastructure-pyti

GSOC first meeting : PyPI Testing Infrastructure (PYTI)

2010-06-03T10:23:00.005+01:00

Hello all,

This my second post in this bloc and my first for the GSOC project : PYPI Testing Infrastructure (PYTI).

The project for the ones that don't know it is about creating a testing infrastructure to test packages that get upload it to PYPI check this link for more information .

For that matter we chose to work with pony-build as the continuous integrating system that will let us run the test in an EC2 machines that run on different platform (linux, windows, unix ...), and as an user interface, we chose to build a web interface with Django that will let the user :

build his own packages in the platform of his choice .
see build report with some other stuff related to the build like a time-line of build ...
every user must have an account in PYTI where he can enable some feature like a notification system for the build , schedule build to be run ...

In this first two weeks i was creating a prototype of the Web interface , now i'm adding some feature on it ,i hope that i will finish this week i will publish some screenshot as soon as i have something ready.

For the next days of this week i have to much task to do :

write a help of how using PYTI.
interface pony-build with the web interface that we created.
automatize the installation of the web interface .
add some feature to the Web interface

for the ones that are interested you can follow us on this link.

2010-04-05T06:04:00.000Z

Hello everybody, and welcome to my first blog and this is my first posting .

If you want to know a little about me you can view my profile (it's not much) .
Now about this blog like the title said it's about python and all the technologies around , if you don't know anything about python it's time to learn you can refer to the excellent tutorial to begin with dive into python. In this blog i will try to write about what i learn or learned while i programme with python and the technologies around and i hope that this blog will be a help for the most of you how want to learn more about python and to admire the beauty of this language .

I will try to talk about a different subject every week related to python and differents other subject like web programming, game programming, mobile programming, software testing and much more that use python .

Finally i will need your help to accomplish this so i will need feedbacks , and the most important thing correct me if i'm wrong don't let me dive in my ignorance.
N.B : you will have to excuse my english i'm not an english native and i try to do my best to write without mistakes.