Initial value in ‘vmmin’ is not finite even when changing the starting value

tl;dr your model needs 4 parameters, but you only gave 3 parameters in the starting vector. I figured this out by (1) trying OLS(c(1,1,1),y=y,z=z) (to confirm that the return value is NA at the starting value); (2) setting debug(OLS) and stepping through it.

When you step through the function checking values as you go you can see that sigma2 becomes NA, because k==3 (the model matrix has three columns), and you only gave three values, so theta[k+1] is one beyond the end of the vector and gives NA (it would be nice if R gave an indexing error in this case, but it doesn’t).

You didn’t give a reproducible example, so I made one up …

set.seed(101)
y <- matrix(rnorm(100), ncol=1)    
z <- cbind(1,rnorm(100),rnorm(100))
OLS <- function(theta,y,z){    
   n <- nrow(z)    
   k <- ncol(z)   
   beta <- theta[1:k]    
   sigma2 <- theta[k+1]    
   e <- y-z%*%beta    
   logl<- -0.5*n*log(2*pi)-0.5*n*log(sigma2)-((t(e)%*%e)/(2*sigma2))    
   return(-logl)    
}

OLS(c(1,1,1),y=y,z=z)  ## NA

On the other hand, this works fine.

OLS(c(1,1,1,1),y=y,z=z)

p <- optim(c(1,1,1,1),OLS, method="BFGS", hessian=TRUE, y=y, z=z)
p
$par
[1] -0.03281533  0.10308645 -0.02229842  0.85335713

$value
[1] 133.965

$counts
function gradient 
      47       16 

$convergence
[1] 0

$message
NULL

$hessian
              [,1]          [,2]          [,3]          [,4]
[1,]  1.171842e+02 -4.922016e+00  2.426181e-01  3.779377e-05
[2,] -4.922016e+00  1.171892e+02  1.468891e+01  3.787193e-05
[3,]  2.426181e-01  1.468891e+01  8.838051e+01 -1.979572e-05
[4,]  3.779377e-05  3.787193e-05 -1.979572e-05  6.866123e+01