RSS

Design notes #1 – Detecting Abrupt Restarts

12 Oct

Theres are  few mods avalilable that can go back to a map after server goes down,
most notably it is built into jcoopz1 , and in the mod dzmapmutator.
both closed source mods.

A while back somone asked if you could detect if the server was told to close , or closed unexpectadly.
Some of use talked back and forth and discussed how dzmapmutator probably handles the situation,
dots provided this simple example of detecting the situation by adding a extra paramenter to the traveling url of the server.

—————————————————————————-

local string S;

S = Level.GetLocalURL();

if (InStr(S,”?restarted=true”)!=-1)
{
// already changed our url
}else{
Level.ServerTravel(ReplaceStr(s, “0.0.0.0/”, “”)$”?restarted=true”,false);
}

—————————————————————————-

What it does is basicly check to see if  a paramenter is present in the servers url , and if it is not ,
we re-navigate to the map addign the argument.

Pretty cool huh?

Knowing this  we can have the server resume at the map that we left off at when the server went down,
This is generally desirable , for the sake of mappacks, or servers with a set map rotation.

to do this we just need to need to store the last map that was loaded.
we can also save the uptime as well

——————————————————————

var config() string crashmap;
var config() int uptime;

local string S;
S = Level.GetLocalURL();
if (InStr(S,”?restarted=true”)!=-1)
{ // already changed our url
crashmap = string(level.outer); // save te3 level to config
uptime ++;
}else{
if  (crashmap ==””) // incase of no stored map
{
Level.ServerTravel(ReplaceStr(s, “0.0.0.0/”, “”)$”?restarted=true”,false);
}else{
Level.ServerTravel(crashmap $”?restarted=true”,false);
}
}

——————————————————————–

What this will do when the server starts , It checks if the paramenter is stored in the url ,
if it is not it then checks to see if we had a last map we were at stored ,
if not the map reloads with the peramaters.if it has a saved map stored , it travels to the last map that was loaded.

Thats the standard way its handled, But what about the crash itself ?
how do we detect if the server was shutdown vs the server exiting improperly?
is it even possible?

sort of.But we need a but of help from outside the game tho.

unreal.exe creates running.ini while the game is running.
if the game abruptly  closes ( gpf  etc) usally this file is not cleaned up.

If you use a batch file to Launch your server, it can easily be modified to detect this remnant.

This is a example batch file that can detect the running.ini at launch

@echo off
rem cls
:restart
if exist running.ini (
echo Abrupt restart!
unreal.exe server SpireVillage.unr?Game=JCoopZ1.JCoopZGame?crash=true Difficulty=3 ini=unreal.ini -timestamplog -server
goto Restart
) else (
echo Server Started Normally
unreal.exe server SpireVillage.unr?Game=JCoopZ1.JCoopZGame?crash=false Difficulty=3 ini=unreal.ini -timestamplog -server
goto Restart
)

We use the existance of the running.ini to detect there was a fault, then  pass that paramenter to the game in the commandline,
so we can be later used in   unrealscript.

the Code to detect the crash in game is very similar to the returntocrashmap code above.

———————————————————————————–

local string S;
local bool serverabrupt;

S = Level.GetLocalURL();

if( InStr(S,”?crash=true”)!=-1 )   // Running.ini was present at launch
{
// abrupt start.

}else{
//non crash start

}
————————————————————————————-

They can both be combined with  some extra bools.
This detects clean exit vs aburt exit, and also when the server goes down for any normal reason,
will resume  where it left off.

————————————————————————————
var config() string crashmap;
var config() int uptime;

local string S;
local bool serverabrupt;

S = Level.GetLocalURL();

if( InStr(S,”?crash=true”)!=-1 )   // Running.ini was present at launch
{

// abrupt start.
serverabrupt = true; this flag is set every map switch  , so any mod can detect it.
if (InStr(S,”?restarted=true”)!=-1)
{ // already changed our url
crashmap = string(level.outer); // save the level to config
log(“saving crash map…”);
uptime ++;
}else{
uptime=0;
if  (crashmap ==””) // Incase of no stored map
{
log(“abrupt crash with No saved crash map… reloading level”);
Level.ServerTravel(ReplaceStr(s, “0.0.0.0/”, “”)$”?restarted=true”,false);
}else{
log(“aburt crash . loading saved crashmap..”);
Level.ServerTravel(crashmap $”?restarted=true”,false);
}
}

}else{
//non crash start

serverabrupt = false; this flag is set every map switch  , so any mod can detect it.
if (InStr(S,”?restarted=true”)!=-1)
{ // already changed our url
crashmap = string(level.outer); // save the level to config
uptime ++;
log(“saving crash map…”);
}else{
uptime=0;
if  (crashmap ==””) // Incase of no stored map
{
log(“First run ! restart with No saved crash map… reloading level”);
Level.ServerTravel(ReplaceStr(s, “0.0.0.0/”, “”)$”?restarted=true”,false);
}else{
log(“first run!. loading saved crashmap..”);
Level.ServerTravel(crashmap $”?restarted=true”,false);
}
}

}
———————————————————————————————

Uptime stats saved in uptime int, it’s reset ever time a serverstart occurs,
also saves serverabrupt bool to note of a abrupt start.

In the rare case of a map that could pottentaily crash repeatady ,

one can try add repeat crash detection.
not exactly sure how one would know to breakout.
we could keep rack of the last crashed map ( technicly we already know the last map).
and keep a tally of how many times it close improperly .

so I came up with this even better mess.
– Detects crash / vs normal start
– Keeps stats of every level that the server crashed on, ever.
– Resumes the last map if it never crashed more then 5 times before.

– important note : code below untested , writtin in notepad.

———————————————————————————————————-

var config() int crashtimes;
var config() string crashmap;
var config() int uptime;

struct maperrors()
{
var string map;
var int crashes;

}

var config() maperrors mapcrashstats[255];
var string S;
var bool serverabrupt;
var int returnvaltemp;

function int trackmapstats(string map,optional bool incement)
{
//lookup a saved map data. return a count
//if its not here add it and return 1
//if you use bool arg “incement” add one to the count for our match when we do the lookup.

local int bc,foundline;
local bool bfound;

For( bc = 0; bc <  255 ; bc++ )
{
if (mapcrashstats.map !=”” && mapcrashstats.map == map)
{
//found map saved
foundline = bc;
bfound = true;

if (incement)
{
mapcrashstats.crashes ++;
saveconfig();
return mapcrashstats.crashes;
}else{
saveconfig();
return mapcrashstats.crashes;

}

}
}

if (!bfound)
{

For( bc = 0; bc <  255 ; bc++ )
{
if (mapcrashstats.map ==”” && !bfound)
{
//open slot
mapcrashstats.map= map;
foundline = bc;
bfound = true;
mapcrashstats.crashes ++; // add out first
saveconfig();
return mapcrashstats.crashes; // break out
}

}
}

}//!found

}

=========================================================================================
// on mutator startup / init / prebeginplay

if( InStr(S,”?crash=true”)!=-1 )   // Running.ini was present at launch
{

// abrupt start.
serverabrupt = true; this flag is set every map switch  , so any mod can detect it.
returnvaltemp = trackmapstats(string(level.outer),true); // lookup or store the map stats
// save  the stats for the last crashed map. , only save at first start.
log (“This map has Crashed ” $ returnvaltemp $  ” timees  in the past”,’Warning’);

if (InStr(S,”?restarted=true”)!=-1)
{ // already changed our url
crashmap = string(level.outer); // save the level to config
log(“saving crash map…”);
uptime ++;
}else{
uptime=0;
if  (crashmap ==”” || returnvaltemp > 5) // Incase of no stored map OR map crashed to many times
{
if (returnvaltemp > 5)
{
log(“Previously loaded map encountered to many crashes… reloading level”);
}else{
log(“abrupt crash with No saved crash map… reloading level”);
}

Level.ServerTravel(ReplaceStr(s, “0.0.0.0/”, “”)$”?restarted=true”,false);
}else{
log(“aburt crash . loading saved crashmap..”);
Level.ServerTravel(crashmap $”?restarted=true”,false);
}
}

}else{
//non crash start

serverabrupt = false; this flag is set every map switch  , so any mod can detect it.
if (InStr(S,”?restarted=true”)!=-1)
{ // already changed our url
crashmap = string(level.outer); // save the level to config
uptime ++;
log(“saving crash map…”);
}else{
uptime=0;
if  (crashmap ==””) // Incase of no stored map
{
log(“First run ! restart with No saved crash map… reloading level”);
Level.ServerTravel(ReplaceStr(s, “0.0.0.0/”, “”)$”?restarted=true”,false);
}else{
log(“first run!. loading saved crashmap..”);
Level.ServerTravel(crashmap $”?restarted=true”,false);
}
}

}

—————————————————————————————————-
so i guess that adreesses everyones concerns. lol

Advertisements
 
5 Comments

Posted by on October 12, 2015 in Uncategorized

 

5 responses to “Design notes #1 – Detecting Abrupt Restarts

  1. Kelly Davis

    October 12, 2015 at 2:15 pm

    I’d like to add this to my server Bob. I run a custom restart batch file that works very well so do I just put the mod code into something like the coop controller and run it in post? BTW you are welcome to copy my server startup, just grab it from the FTP. It’s time/datestamped and a bit more thorough than yours.

     
    • bobisunreal

      October 12, 2015 at 9:24 pm

      Yeah should be able to just add it to any running class / mutator.
      When the server starts the first map it will re-switch to the current map to add the arguments tho.

      I forgot to include info on some extra code in case it detected as a crash,
      it could go back to the level it was last on when it crashed ( probably easy for you to add)
      just save the mapfilename to config after “server started up “normally”/ logger tagged.
      then grab it in the case of the 2 non tagged conditions. using something like
      “servertravel %savedmap%??restarted=true” instead of the 0.0.0.0 lines.

      also i omited the code nessecally to save up time to config file , since i used a alternate way to save the data, but that also very easy to add.

      also the argument -timestamplog automatically sets the log file name to the current date/time s you dont need to mess with the get time shit in the batchfile.

      … hmm all these points was suppose to be talked about in this article,
      that was the entire point of the article ,
      I think i am going to have to re write the whole post now.
      i so half ass everything in my rush to say things out loud..

       
      • Kelly Davis

        October 14, 2015 at 12:26 pm

        You ought to consider a breakout scenario if you write it to try to return to the map that crashed. I get random crashes now and again but the majority of hard server crashes are problems that need addressed and keeping an eye on the logs lets me do that. I’d def consider a rewrite of this since it’s super important to people who care about server health. I’ll be including this in my next coop update so I’ll let you know when it goes on so you can see the logs.

         
  2. bobisunreal

    October 17, 2015 at 9:14 pm

    ok i re wrote the whole article. it addresses those concerns , and provides a example how logs crash stats for every map, and if the map crashes too often , wont resume from the bad map.
    i actually think thats even more handy , since you can check crash stats for every map in the ini

     

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s