FRIHOST FORUMS SEARCH FAQ TOS BLOGS COMPETITIONS
You are invited to Log in or Register a free Frihost Account!


Cronjob Question :)





Aredon
Let me start by first explaining what I plan to do here. I run a website for my guild, and recently I began coding support for XML data pulled from The Amory. Mostly for character profiles, but also for an overall roster of members. Currently when you visit a user's profile it checks the cache for data, then the timestamp on the cached file, then it pulls the XML data from either the armory or the cache depending on the age of the file. Now, this is all fine and good, and I'm actually very happy with the functionality, but when we get into the roster I really can't be pulling 60+ XML files at a time from armory. It uses too much of their bandwidth and they will ban my IP for 24 hours. Sad So what I was planning on doing was making a cronjob that ran every few days, and ran queries one at a time (most likely with sleep()) from armory. However, this would mean that every few days I would be running a cronjob for potentially several hours. So I was basically just curious if that was going to be a huge problem before I go and do it. I'd hate to cause too much server load. Wink
rvec
Why not just let cron run every two hours. That way you won't have a script sleeping, and you can be sure only one pull is done every two hours (or more depending on how much you can and need to).
Aredon
rvec wrote:
Why not just let cron run every two hours. That way you won't have a script sleeping, and you can be sure only one pull is done every two hours (or more depending on how much you can and need to).
Well it doesn't need to be two hours apart, armory only requires that you pull data no faster than someone would browse. So one per minute works fine, and I'm not really sure how I feel about doing it every two hours. Mostly because we currently have somewhere around 80 characters in the database, and probably several more that haven't been cached yet. I would like the data updated every 4 days, 80 characters at 1 per 2 hours is about a week, and the more characters we get in the database the longer that would stretch. Thereby changing the cache age to a function of the number of characters, and I was kind of hoping this would be a constant. Wink (or as close as I can get to a constant)

This is basicaly what the code would look like for each character. (missing one or two queries and a file write for the cache) Shouldn't be too terribly stressful on the server to run one at a time at a steady pace, but I wanted to make sure.
Code:
define("USE_CURL", true);
      $realm = "Hyjal";
      $url = "http://www.wowarmory.com/character-sheet.xml?r=" . $realm . "&n=" . $name ;
      $ch = curl_init();
      $useragent="Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.1) Gecko/20061204 Firefox/2.0.0.1";
      curl_setopt ($ch, CURLOPT_URL, $url);
      curl_setopt ($ch, CURLOPT_RETURNTRANSFER, 1);
      curl_setopt ($ch, CURLOPT_USERAGENT, $useragent);
      $f = curl_exec($ch);
      curl_close($ch);


EDIT: I was kind of unclear here, the cronjob would only be updating cache data where it was expired (over 4 days).
rvec
hmm I think you could run that. Maybe it would even be better to do it like this instead of running 60 in a couple of sec.

Just make sure to use functions, classes and unset to make sure you don't exceed the memory limit (by putting all the curl returns in an array for example).
Aredon
rvec wrote:
hmm I think you could run that. Maybe it would even be better to do it like this instead of running 60 in a couple of sec.

Just make sure to use functions, classes and unset to make sure you don't exceed the memory limit (by putting all the curl returns in an array for example).

The curl returns are getting saved to an XML file.

EDIT: ok I have the draft of my updater ready. I was able to bypass the need for a loop entirely. Which should have been very obvious to me before. :/

current Cronjob: (to run every two minutes until no longer needed, every 4 days)
Code:

define('WITHINPHP', 1);
include 'EDITED OUT :)';
$db=new DBManager();
$db->connect();
include '/home/aredon/domains/clanhunter.frih.net/public_html/navigation/modules/armory.php';
     
$t=time()-60*60*24*4;
$sql="SELECT * FROM `armory_cache` WHERE `timestamp` <= '".$t."' ORDER BY `timestamp` ASC LIMIT 1";
$result=mysql_query($sql);
if($row=mysql_fetch_object($result)){
     pull_char_armory($row->char);
}else{
     exit;
}


the armory function
Code:

function pull_char_armory($name){
   define("USE_CURL", true);
   $realm = "Hyjal";
   $url = "http://www.wowarmory.com/character-sheet.xml?r=" . $realm . "&n=" . $name ;
   ## Start cURL
   $ch = curl_init();
   $useragent="Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.1) Gecko/20061204 Firefox/2.0.0.1";
   curl_setopt ($ch, CURLOPT_URL, $url);
   curl_setopt ($ch, CURLOPT_RETURNTRANSFER, 1);
   curl_setopt ($ch, CURLOPT_USERAGENT, $useragent);
   $f = curl_exec($ch);
   curl_close($ch);
   ## Finish cURL
   if(!$xml = new SimpleXMLElement($f)){
      return FALSE;
   }else{
      if(!$xml->characterInfo->characterTab->characterBars->health['effective'])return FALSE;
      char_xml_write($f,strtolower($name));
      return $xml;
   }
}


The XML write function
Code:

function char_xml_write($xml,$char){
   $filename='/usr/home/aredon/domains/clanhunter.frih.net/public_html/navigation/cache/armory/'.$char.'.xml';
   $fh=fopen($filename, 'w') or die("can't open file");
   if(fwrite($fh, $xml)){
      $stuff=true;
      $sql="SELECT * FROM `armory_cache` WHERE `char`='".mysql_real_escape_string($char)."'";
      $result=mysql_query($sql);
      if(!$row=mysql_fetch_object($result)){
         $sql="INSERT INTO `armory_cache` (`char`,`timestamp`) VALUES ('".mysql_real_escape_string($char)."','".time()."')";
         @mysql_query($sql);
      }else{
         $sql="UPDATE `armory_cache` SET `timestamp`='".time()."' WHERE `char`='".mysql_real_escape_string($char)."'";
         @mysql_query($sql);
      }
   }
    fclose($fh);
   return $stuff;
}
Related topics
Reply to topic    Frihost Forum Index -> Support and Web Hosting -> Web Hosting Support

FRIHOST HOME | FAQ | TOS | ABOUT US | CONTACT US | SITE MAP
© 2005-2011 Frihost, forums powered by phpBB.