Sahi for performance testing?

juhanay

Hi
I am new to sahibut it looks nice. I need a software to measure the performance of a rails project for regression testing purposes. I need to test the web-appliction with different browsers (e.g IE,Chome, Firefox). I have been wondering about using sahi for this purpose for recording different tests and measuring the time how long it takes to run them (run the tests few times and then calculte average time perhaps).

I plan to use similar method as described in the forum post id=625
to calulate the elapsed time. Developers can then use this data to make application better suited for some browsers, after they know what actions are slow in some browser. After some changes tests would be run again to see how well they succeeded. How suited sahi is for this purpose in your oppinion?

Also second question what would be the best way to store elapsed time information so that it could be easily visualisable, for example with google charts. I noticed that sahi logs have some time stamps but they do not contain the total elapsed time. Also they do not contain web-browser name (e.g. IE,Firefox, etc). I would like to modify the reports somehow since at least initally i would use sahi only for performance testing.

pankaj.nith

Hi,

I hope features like multithreaded support and suite/batch run will also be useful for performance testing. Please discuss further how you plan to use sahi.

Regards,
Pankaj

juhanay

I was planing to do a simple start. For example i could record a set of pre-defined actions with the recorder and play them as a batch. I could manually select some actions/cases that tests some feature speed. Then i could record like 10-20 tests that test different aspect of the system. Tests are not meant to catch bugs just to measure how long actions/cases take. These batches would be run automatically in different browsers(IE,Firefox, Chrome, Opera etc.) and the overall time would be recorded in a log file. Then i would need somekind of script that would collect this data from the logs and make e.g. a web page with google chart visualization. This is just to give developers an easy accesss to the data. A small amount of eye candy does not hurt either.

I could even calculate somekind of performance index for the system using this data. Simplest way to calculate the index would be using the elapsed time.

The goal would be to give this performance data to the developers and they could see how their changes affect the system performance. This could be put as a part of build cycle and under cruise control. Also when we have a some reference point we can then tell how good job we are doing when we are trying to improve speed on some browsers. Tests would be run by the same machine so the hardware would not affect the tests so much. Perhaps a seperate performance test server is needed to make sure that no one is slowing the server during the tests. This is still under planing and I welcome any suggestions and advices that you can offer.

I made a quick demo with sahi and it worked fine. I would need to improve reporting part a bit. The biggest advantage of sahi as i see it that it actually runs the sript in a different browser and it does not emulate a browser. Selenium could be another candicate for this but it seems that sahi is simpler to use.

narayan

Juhanay,

There is a hidden function in Sahi which gives you details about browser etc.

Try

_log(_sahi.getDiagnostics());

or

var $diagnostics;
_set($diagnostics, _sahi.getDiagnostics());
_writeFile($diagnostics, "D:/your/file.txt");

Not sure how you would go about getting the time diffs. You could do it with some small program which reads the log files and gets the data, or you could modify Sahi's code itself so that the logs automatically have the required data. Let us know if you have a definite structure of required data in your mind, and we can add/modify Sahi to be able to generate such logs.

A couple of points that you need to keep in mind:

1) Sahi adds javascript via the proxy which will make pages load slower than they would without Sahi. So a comparison of two runs will be meaningful but a single run will not be proof of application performance.

2) There is a fixed gap of 100 ms between steps in Sahi. You can change this value in sahi.properties but it is still means there is a granularity beyond which you cannot differentiate performance. This should not bother you too much since web application delays in loading etc. are at least an order of magnitude higher.

Regards,
Narayan

Rohini

Hi Narayan,

I was looking at the last post wherein you have mentioned that the sahi code itself could be modified to have the time taken for executing the script. I created a link called Time difference in the controller7.htm and on accessing the link i get a classNotfoundException. Would it be possible to brief on this aspect to use sahi for performance testing.