Converting from a hexadecimal string to a number and vice-versa can be done using the built-in TestStand Functions.
The following code could be used to convert a hex string to a number, increment that number and then convert it back to a hex string:
locals.Num = Val("0x" + locals.String), locals.Num ++, locals.String=Str(locals.Num, "%x")
The increment portion of the code (Locals.Num ++) could be replaced with any mathematical function that needed to be performed on the value.