18 January 2026:


This forum is now archived and is in read-only mode. Please continue discussions on our improved new Sahi Pro Community forum.



Sahi Pro is an enterprise grade test automation platform which can automate web, mobile, API, windows and java based applications and SAP.

_assertContainsText() on robots.txt

globalwormingglobalworming Moderators
edited November -1 in Sahi - Open Source
Hi guys

that's the problem, I have to make sure a specific string is in the robots.txt for a website. I find no way to access this file with Sahi, does anyone can up with a way to get this text. If not with Sahi, maybe with a shell script?

Regards
Wormi

Comments

  • _dkulkarni_dkulkarni Members
    Hi Wormi,

    You could download robots.txt (Click "Configure" on the dashboard-> download_urls.txt-> add the url in download_urls.txt -> Save-> Restart Sahi).
    You can then read the file with the _readFile API.
    Would that help?

    Regards,
    dkulkarni
  • globalwormingglobalworming Moderators
    Thanks for your response..

    I added the URL/robots.txt in sahi/config/download_urls.txt and to be sure, in sahi/userdata/config/download_urls.txt as well :) Then restarted the Sahi Proxy and tried $text= _readFile($URL+"robots.txt) and got a FileNotFound Exception. Do you know where this file gets downloaded to? I Or am I just doing it wrong?

    Regards Wormi
  • SumitraSumitra Members
    Hi globalworming,

    You will have to use regular expression to add the URL/robots.txt.

    Add .*robots[.]txt in Sahi/userdata/config/download_urls.txt.

    Regards
    Sumitra
  • globalwormingglobalworming Moderators
    Thanks, I'll try it when my testsuite finishes^^
  • globalwormingglobalworming Moderators
    edited August 2011
    I don't get it.. I added the line as you said, but keep getting this exeption. 
    Wrapped net.sf.sahi.util.FileNotFoundRuntimeException: java.io.FileNotFoundException: /home/wormi/workspace/mt-test-szt/scripts/http:/sxxxxxxxxxx.eu/robots.txt (No such file or directory)

    Please help :)
  • _dkulkarni_dkulkarni Members
    Hey Wormi,

    Adding .*robots[.]txt in download_urls.txt will download the file in sahi/userdata/temp/download/
    This text file will have a session id attached to it.

    To read this file, you can use something like this: 
    var $downloadedFile = "robots.txt";
var $filePath = _userDataPath($downloadedFile);
_navigateTo("http://site.com/robots.txt");
_assertEqual($downloadedFile, _lastDownloadedFileName());
_saveDownloadedAs($filePath);
_readFile($filePath);

    Regards,
    dkulkarni
  • globalwormingglobalworming Moderators
    Hmm... when I use this code like you said, the controller can't execute the assert and after a whjle, my PC gets stuck.. when I close the controller, Sahi just mentions: Rhino lib:Step >_sahi._assertEqual("robots.txt", _sahi._lastDownloadedFileName());< did not complete in 150 seconds
  • SumitraSumitra Members
    Hi globalworming,

    Can you please post the script?

    Regards
    Sumitra
  • globalwormingglobalworming Moderators 
    $file = "robots.txt";
$savePath=_userDataPath($file); 
_navigateTo($startURL+$file);
_assertEqual($file, _lastDownloadedFileName());
_saveDownloadedAs($savePath);
_log(_readFile($savePath));
  • narayannarayan Administrators
    There is a much simpler way.

    Instead of adding it to download_urls.txt, just do

    var $robotsTxt = _readURL("http://yoururl/robots.txt";); // reads the text
    _assertTrue($robotsTxt.indexOf("specificstring") != -1);

    Regards,
    Narayan
  • globalwormingglobalworming Moderators
    works well!

    thank you all for your help.
This discussion has been closed.