_assertContainsText() on robots.txt

globalworming

Hi guys

that's the problem, I have to make sure a specific string is in the robots.txt for a website. I find no way to access this file with Sahi, does anyone can up with a way to get this text. If not with Sahi, maybe with a shell script?

Regards
Wormi

_dkulkarni

Hi Wormi,

You could download robots.txt (Click "Configure" on the dashboard-> download_urls.txt-> add the url in download_urls.txt -> Save-> Restart Sahi).
You can then read the file with the _readFile API.
Would that help?

Regards,
dkulkarni

globalworming

Thanks for your response..

I added the URL/robots.txt in sahi/config/download_urls.txt and to be sure, in sahi/userdata/config/download_urls.txt as well Then restarted the Sahi Proxy and tried $text= _readFile($URL+"robots.txt) and got a FileNotFound Exception. Do you know where this file gets downloaded to? I Or am I just doing it wrong?

Regards Wormi

Sumitra

Hi globalworming,

You will have to use regular expression to add the URL/robots.txt.

Add .*robots[.]txt in Sahi/userdata/config/download_urls.txt.

Regards
Sumitra

globalworming

Thanks, I'll try it when my testsuite finishes^^

globalworming

I don't get it.. I added the line as you said, but keep getting this exeption.

Wrapped net.sf.sahi.util.FileNotFoundRuntimeException: java.io.FileNotFoundException: /home/wormi/workspace/mt-test-szt/scripts/http:/sxxxxxxxxxx.eu/robots.txt (No such file or directory)

Please help

_dkulkarni

Hey Wormi,

Adding .*robots[.]txt in download_urls.txt will download the file in sahi/userdata/temp/download/
This text file will have a session id attached to it.

To read this file, you can use something like this:

var $downloadedFile = "robots.txt";
var $filePath = _userDataPath($downloadedFile);
_navigateTo("http://site.com/robots.txt");
_assertEqual($downloadedFile, _lastDownloadedFileName());
_saveDownloadedAs($filePath);
_readFile($filePath);

Regards,
dkulkarni

globalworming

Hmm... when I use this code like you said, the controller can't execute the assert and after a whjle, my PC gets stuck.. when I close the controller, Sahi just mentions: Rhino lib:Step >_sahi._assertEqual("robots.txt", _sahi._lastDownloadedFileName());< did not complete in 150 seconds

Sumitra

Hi globalworming,

Can you please post the script?

Regards
Sumitra

globalworming

$file = "robots.txt";
$savePath=_userDataPath($file); 
_navigateTo($startURL+$file);
_assertEqual($file, _lastDownloadedFileName());
_saveDownloadedAs($savePath);
_log(_readFile($savePath));

narayan

There is a much simpler way.

Instead of adding it to download_urls.txt, just do

var $robotsTxt = _readURL("http://yoururl/robots.txt"; // reads the text
_assertTrue($robotsTxt.indexOf("specificstring") != -1);

Regards,
Narayan

globalworming

works well!

thank you all for your help.