Accessibility: QA Meetings 2008 Aug 13

10:15 <bgmerrell> are either of you familiar with the term “smoke test?”
10:15 <calen> have any adjustment?
10:16 <ray> do i need a cigarette? joking :P
10:16 <bgmerrell> hehe
10:16 <calen> i hear that, but i don’t know how to do
10:16 <ray> does it means stress test?
10:16 <bgmerrell> okay,… first of all..
10:16 <calen> heard
10:16 <bgmerrell> from Wikipedia “Smoke testing is done by developers before the build is released or by testers before accepting a build for further testing.”
10:17 <bgmerrell> we will do the second part, i.e., QA will perform a smoke test before accepting a build for further testing.
10:18 <bgmerrell> it’s like a sanity test.
10:18 <bgmerrell> the idea will be for one of us to validate that new build and/or RPMs are not broken
10:19 <bgmerrell> and that we can all test effectively with the new RPMs before we all try to download and use them
10:19 <bgmerrell> does that make sense?
10:19 <calen> bgmerrell: yes
10:19 <ray> yes
10:19 <ray> actually, calen already did it :)
10:19 <calen> bgmerrell: that is what i had said in our test plan befor i think:)
10:19 <bgmerrell> excellent :)
10:20 <calen> ray: not exactly :)
10:20 <bgmerrell> calen: do we have any winforms tests that are 100% finished and not waiting for some bug?
10:20 <calen> bgmerrell: no
10:20 <bgmerrell> okay, i didn’t think so
10:21 <bgmerrell> when we do, we will want to make sure it is added to the test suite (tests.py)
10:21 <bgmerrell> then we can start doing our smoke tests
10:22 <bgmerrell> then we will designate one person to download the newest RPMs and update a wiki page to inform others if they should update or not
10:22 <calen> bgmerrell: do you mean use our strongwind test code to do smoke tests
10:22 <calen> ?
10:23 <bgmerrell> calen: for now yes, if they start to take too long, maybe we will have a smoke-tests.py to perform only a few tests
10:23 <bgmerrell> instead of tests.py which will have all the tests
10:23 <ray> calen, i think bgmerrell means we could download the rpms which is newly built, and install them to see if there are some prolbems
10:24 <bgmerrell> ray: right, but to see if there are some problems we can use our strongwind tests
10:24 <ray> ah , o k
10:24 <bgmerrell> if the most/all of the tests fail, when they were successful before, we know there are problems :)
10:24 <bgmerrell> calen: does that sound okay? or did you have a different idea?
10:25 <calen> bgmerrell: but my idea is if we should check if all patches which <decriptor> want to build into have been included, and all rpms can be install the first, than run our test
10:25 <bgmerrell> calen: yeah, sounds perfect
10:25 <calen> then to make sure new rpms is ready
10:27 <bgmerrell> How would we “check if all patches which <decriptor> want to build into have been included?”
10:29 <calen> i think it need decriptor give us a list
10:30 * ray is looking for some details about smoke test
10:30 <calen> and extra rpms to give a diff :)
10:31 <calen> i don’t know if it can be done like that \^\^
10:32 <bgmerrell> http://build1.sled.lab.novell.com/uia/ has previous rpms
10:32 <bgmerrell> if we know the revision numbers that were used to create the RPMs we can look at the ChangeLog differences
10:32 <calen> sorry, i mean get patches from rpms
10:32 <ray> i got it, smoke test is something like sanity check, then people could do the next
10:33 <bgmerrell> ray: right, http://en.wikipedia.org/wiki/Smoke_test#Smoke_testing_in_software_development
10:34 <calen> that is name “?????” in china. hehe
10:36 <calen> bgmerrell: sorry, there is no way to check out patches from rpm :(
10:37 <calen> how can we know new patch has been include into rpm?
10:38 <ray> bgmerrell, so when we install new rpms and no one is broken, then we can test in strongwind?
10:38 <bgmerrell> calen: the only thing we can do is look at the changelog
10:38 <bgmerrell> decriptor would have to do the same thing
10:39 <bgmerrell> changelogs*
10:39 <ray> calen, i think best way is to see the rpm’s changelog or revision
10:41 <calen> according changelogs to run samples which has been mentioned in changelogs, then run our strongwind test?
10:41 <bgmerrell> i’m trying to think about the changelogs.. because there will be a lot of changelogs to look at
10:42 <bgmerrell> i think our bugs will be the best indicators
10:42 <calen> because sometimes some control’s application cann’t be invoked after update the rpms
10:42 <ray> bgmerrell, what about to see the rpms’s changelog ,
10:42 <bgmerrell> ray: where is that?
10:43 <ray> rpm -q –changelog, do we talk about the one thing ? :)
10:43 <ray> rpm -qp –changelog xxx.rpm
10:44 <bgmerrell> let me download the new RPMs
10:45 <calen> there is no changelog by use rpm -qp –changelog ***.rpm now
10:46 <bgmerrell> okay
10:46 <bgmerrell> I will talk to jared and decriptor about this
10:46 <bgmerrell> and see what we can do
10:46 <bgmerrell> we don’t have to solve it now
10:46 <calen> it return (none), maybe descriptor will add it ??
10:47 <bgmerrell> the problem is we have lots and lots of rpms
10:47 <bgmerrell> I think the best thing to do would be to have the builders add a comment to a bug when its patch has been built
10:48 <bgmerrell> does that sound reasonable?
10:48 <bgmerrell> i think that is how other teams do it
10:48 <bgmerrell> but i will need to discuss that with jared and stephen
10:49 <calen> but they also would add something new except for bugs
10:50 <bgmerrell> yes, true, but is it necessary to monitor *everything* in each rpm?
10:50 <bgmerrell> the wiki pages are supposed to abstract that for us so we can see the progress
10:53 <bgmerrell> i don’t think it is necessary, but maybe i’m dumb :)
10:53 <calen> i still have problem:( i think the best way is from builder, because just builder have the exactly idea which has been build
10:54 <bgmerrell> i don’t know anything about building.. does he know exactly what is being built?
10:54 <bgmerrell> maybe ray knows
10:54 <calen> if a developer make ‘done’ for one control or one bridge, but builder never include, we also can’t do test for it
10:56 <bgmerrell> but the builds are created at each revision automatically
10:56 <ray> when devs commited the code, svn will add a revision number, then the build machine will build all the source code i think
10:56 <bgmerrell> so any code that is checked it *should* be built
10:56 <calen> oh..that make sense
10:57 <bgmerrell> so maybe the developers just adding the revision number when they mark a bugged as fixed will be good enough
10:57 <bgmerrell> and we can make sure that revision is built
10:57 <bgmerrell> does that sound okay?
10:57 <calen> great
10:57 <bgmerrell> okay awesome, i will talk to knocte about that
10:58 <bgmerrell> but i also wanted to talk about who should do the smoke tests…
10:59 <bgmerrell> i think ray is a good candidate because he is also involved with the builds so he can troubleshoot any problems that are build related
10:59 <ray> no problem
10:59 <bgmerrell> and also he will allow the full-time QA people (calen and I) to focus on testing
11:00 <bgmerrell> what do you think
11:00 <bgmerrell> Calen?
11:00 <calen> agree
11:00 <bgmerrell> great
11:00 <ray> so when rpms updated, i download all of them, install them to see if there are some broken one? or how to do that ?
11:00 <calen> before i join novell, builder do this job in my before company…
11:01 <calen> then he give our QA a test list..hehe
11:01 <bgmerrell> calen: cool, on my previous team the QA automation team did it
11:02 <ray> hehe, you work for Dale?
11:03 <bgmerrell> ray: yeah.. when rpms are updated, you will (1) install them and make sure they all install properly (2) run the smoke tests using the test harness
11:03 <bgmerrell> for now the smoke tests will just be tests.py, but that might change
11:03 <calen> because builder should check all of the patches has been commit. sometimes developer say they have commit a patch in bugzilla but actuality they lost
11:03 <bgmerrell> but we don’t have any tests ready yet
11:04 <bgmerrell> ray: yeah, he was me previous boss :)
11:04 <ray> bgmerrell, just run tests.py
11:04 <ray> just run that script ?
11:04 <bgmerrell> ray: well, you will run local_run.py or remote_run.py, which rely on tests.py
11:05 <bgmerrell> ray: do you have extra test/lab machines in china?
11:05 <ray> yeah, i think i can use my desktop :)
11:05 <calen> bgmerrell: yes, after they fix app closing issue, we can add form test into tests.py
11:06 <bgmerrell> it would be good to run a test on each platform
11:06 <bgmerrell> suse, ubuntu, and fedora.. but maybe we lack hardware
11:06 <bgmerrell> you could run them in Provo and view the logs
11:07 <ray> bgmerrell, for now, we only have suse package from monobuild :)
11:08 <bgmerrell> ray: right, but for the future
11:08 <ray> bgmerrell, oh exactly
11:08 <bgmerrell> ray: maybe in the future i will set up smoke test vms in provo
11:08 <bgmerrell> and you can just run your tests and look at the logs
11:09 <ray> bgmerrell, no matter, i have those VM installed :) , i can test them for now :)
11:09 <bgmerrell> for now you can just make sure the RPMs install, and you can just check to make sure some of the controls that are marked “done” appear in Accerciser
11:10 <ray> ok
11:10 <ray> that’s clear
11:10 <bgmerrell> but hopefully Calen’s form test can be put into tests.py soon!
11:10 <calen> hehe..i hope
11:10 <ray> nice :)
11:11 <bgmerrell> ray: then you can just send out an e-mail to the mailing list
11:11 <bgmerrell> ray: Smoke tests pass! | Smoke tests fail! :)
11:11 <calen> it can’t be run but can’t be exited
11:12 <bgmerrell> yeah, so ray could even run it manually if he wanted
11:12 <bgmerrell> when do you both go back to the office?
11:12 <ray> ah, sound interesting :)
11:12 <calen> Aug.26
11:13 <bgmerrell> calen: do you understand how local_run.py and remote_run.py work?
11:13 <ray> 25 i guess
11:13 <calen> 25, yeah
11:13 <ray> oops, from 25th, we have hack week
11:13 <calen> bgmerrell: i think i understand
11:13 <bgmerrell> calen: have you tried running remote_run.py before?
11:14 <calen> bgmerrell: i just have tried one or two times
11:14 <bgmerrell> calen: cool :) maybe you can show ray how they work when you get back to the office
11:15 <calen> bgmerrell: np :)
11:21 <bgmerrell> ray: so when you see an e-mail from stephen you can try to do smoke tests immediately, depending on the time of day of course :)
11:22 <ray> bgmerrell, sure
11:26 <calen> i have sent test howto to ray, it would be more helpful to him yet
11:26 <bgmerrell> it looks like the developers are already adding the revision number to comments when they fix the bugs
11:26 <bgmerrell> https://bugzilla.novell.com/show_bug.cgi?id=412206
11:27 <bgmerrell> https://bugzilla.novell.com/show_bug.cgi?id=411345
11:27 <bgmerrell> https://bugzilla.novell.com/show_bug.cgi?id=416602
11:27 <calen> yeah..i noticed it
11:27 <bgmerrell> so we can just make sure those revisions are built if we need to
11:27 <ray> yep
11:27 <bgmerrell> calen: great, thanks calen (for sending test howto to ray)
11:31 <bgmerrell> i forgot
11:31 <bgmerrell> one more thing
11:31 <bgmerrell> i already talked to Calvin about this, but i don’t think we will use testopia
11:31 <bgmerrell> testopia is used to track test cases
11:32 <ray> oh due to we have svn ?
11:32 <calen> yeah
11:32 <bgmerrell> but i think we can consider each of our test scripts as test cases
11:32 <bgmerrell> what do you think calen?
11:32 <calen> i agree
11:32 <bgmerrell> i think testopia will create too much “overhead”–additional work
11:32 <bgmerrell> okay, awesome.
11:33 <bgmerrell> we will need to update our test plan accordingly
11:33 <bgmerrell> i can do that