[CentOS-devel] [QA] Test-scripts for Dovecot POP3/IMAP

Hi,

Thanks a lot for your submissions, i'll integrate them in the scripts
repository.
One simple remark : we launch the tests on a minimal setup (from a
minimal ks) so the first thing to do in the script itself is to install
the required package, verify that it installs correctly and then test
the feature that you want to test.

I'm working on redesign quickly something that can be used for the 6.0
tests (now that 5.6 is out of the door)

Fabian

Fabian,
Would you mind opening the process slightly? I mean it'd be helpful
having the test list to avoid people doing the same thing. Also it would
really help to codify the return values, etc.
This is something that should be written on Wiki page, so that people know:
- what are the current scripts
- expected return values
- testing environment and requirements
- etc.
Thanks,
DH

Do you have access to the Wiki?

Hi David,

Sorry for the late answer but due to some illness on my side (and so
being very tired when going back home from $work) i wasn't able to
answer that question. Yes it's really a need to open that part of the
tests so that people can contribute to those tests and also open the QA
process (and also the status) to the community

Yes, and so first a need to design it before documenting it. Let me
clarify this a little bit : actually we (Karan and myself) designed
quickly something that could be used directly during the 5.6 process (we
did that during Fosdem, between discussions with people at the CentOS
booth ...)
We started 'thin' and so actually it's just a file with different
functions declared and some bash scripts (one per VM/type) sourcing that
first file and using the functions.
Those 'functions' contain CentOS QA hardware informations (like on which
machine to deploy a vm, which type of vm (xen, kvm, virtualbox, etc ...)
disk-size, memory etc ..

So we'd like first to separate maybe those informations from the tests
themselves. The tests scripts would be available in a git repo and we'd
use them during QA and post the results somewhere.

I would like to redesign our QA subnet/vlan a little bit and reflect
that in our existing QA script. (I don't think that this part interests
the Community as it also involves public ip, etc ...)

We'd try to have a small meeting with KB about that part and then a
public discussion (an IRC meeting maybe) for people wanting to
contribute their tests.

Part of the discussion would be also about what to test. For example,
testing if dovecot imap/pop3 is working is good, but my personal
approach would be to test if centos packages for example are also
'centosified'.
Let me explain that with a real example : if we have to change a logo
(and that we use the same logo during the whole distro lifetime) a basic
script would be to retrieve that logo, compare it with the known md5sum
and so guess if the logo was replaced or not correctly. Stupid example,
but believe me it's worth writing such stupid script :-)

Fabian

Sorry for jumping into this thread. I'm making it my personal mission to
learn as much as I can about CentOS. Please excuse this noob question...
What is the purpose of these scripts?

The above only further deepens the mystery for me! I want to learn!

Hmmm -- I don't mind waiting, I'm trying to hold onto my enthusiasm
here. I can guess, or try and piece together an answer for myself, but
just some inkling that I was noticed, or something.

I wondered the same thing so i channeled my inner "misc" and opened up
the tests in my favourite text editor.
They test for basic functionality
ie create a basic configuration, switch it on and check that it is
working and give an exit status of pass or fail.
The conversation above suggests that it would be gold standard for the
test for each service to be able to successfully install the service
onto a minimal set up as well rather than having separate install and
test scripts.

open them up and have a look :)

regards

mike

I understand the phrasing of my question assumed that I hadn't done that
- I did the same, and when I noticed the purpose of the scripts, and
what they do, I wondered why is this being "repeated"
This I understand - what I would like to know, is this a new thing? Or
is this part of the normal procedure? Are these scripts uploaded to a
SVN repo where we can get to it, and test against our own builds if we
are so inclined? etc etc
If this is standard - then I'm surprised that the distribution mechanism
is the list. (I'm sure I'm wrong, but understand, this is all I can see
as a newbie, or even someone who was just browsing to see how projects
are run) - I just assumed that the devs used automated script deployment
mechanisms to do this kind of thing.

I'd use nagios as a front end, with NRPE on my target machine with a
minimal install, and push the scripts from an SVN repo to the test box,
and then look in nagios to see what the results of the script is. That
way, I know that when I get green across the board - Everything is all
good!

Way ahead of you! hehe!


I can see what the tests do, I am asking why are we doing these tests?

I'm assuming that this is done as part of the mechanism to test that
certain packages are working as expected after install, once they have
passed binary compatibility. (I would assume that if they are binary
compatible, that they will behave the same way as the RHEL packages)
which if this is the case, then why are we testing them again? Which
leads me to think my assumption is wrong - SO .... why are we doing
these tests? :)

I use nagios to run scripts and get results from environments, along
with other deployment methods to create clean environments to do this
automagically. Does the centos devs do something like this, (I know I
inferred this earlier, bare with me).

Something like this - the writing of the scripts, can that be opened up
a bit? Maybe thrown open to the community? Get them cracking on the
current release, see what comes back - it's just their time and effort,
you just have to ask for it.

Should I be starting a new thread for this by the way, I know how fickle
people can be!

Thanks for the response Mike!

The CentOS Project is building a suite of automated tests that we are
using to help us with QA after we get our tree ready.

This test suite will run on our tree after we get it moved to our QA
servers.

This is being run by people other than Karanbir, Tru, and me ... so it
is already "opened up".

As to how it is being run, that is up to the people who we have "opened
it up to".

See this here? This should be documented and on the web in a very
prominent position. This is awesome to know this.

There are thousands of people who use CentOS who build automated testing
suites, using numerous tools and methodologies.

So, so far I've learnt that we use Mock to build the centos packages,
and we actually do have a test suite in development.

Great - Now, we just needs those two steps up, online, and for the
community to see. Hopefully with hundreds more eyes on it, we can grow
even more. Personally, I think I'll start helping you guys with your PR,
because getting info feels like getting blood from a stone. Call me
impatient of whatever you wish, (not aimed at Johnny Huges or anyone
person,) I will continue to pester you guys, and you can hate seeing
Mister IT Guru in your inbox!! But I truly believe CentOS needs to come
into the light - don't want to CentOS to go the debian root (geddit!)

Currently, I'm using an small install script, before running the
test-script, this will remove some redundancy in test-scripts for the
same service example:

0_install_dovecot.sh

-------------------------
#!/bin/sh
yum -y install dovecot
chkconfig dovecot on
service dovecot start
-------------------------